Communicating an analytics finding is the work of writing a result so that somebody who was absent for all of it can act on it correctly.
A result that has survived its validity checks and been read against a practical threshold exists inside one person's head and one notebook. The decision it supports belongs to a group, and that group includes an engineering manager sequencing next quarter, a finance director who will see the number in a plan and a support lead who has to prepare for the change. None of them will read the notebook.
What usually travels instead is a chart in a message thread, a screenshot of a dashboard or a sentence saying the test won. Each of those loses the part of the result that determines what to do, which is the size of the effect, the uncertainty around it and the threshold it was judged against.
The sections below set out what separates a finding from a chart, give the four parts a written result carries, work one result through in both the weak form and the strong form, and close on the record that outlives the decision.
What separates a finding from a chart
A chart shows a shape. A finding states a claim about the world and says what follows from it. The difference is one sentence, and the sentence is what a reader remembers a week later.
Cole Nussbaumer Knaflic argues the point in Storytelling with Data, published in 2015, and her prescription is direct. The chart is stripped to the single comparison it exists to make, and the claim goes in the title, so that the reader arrives at the conclusion the analyst reached. A chart titled Signup conversion by variant asks the reader to do the analysis again. A chart titled The shorter form gained between 0.7 and 8.2 per cent has done it for them.
The same rule applies to the document around the chart. A section headed Results that contains six figures and no sentence has moved the analysis to another person's desk without moving the conclusion.
The four parts of a written finding
A finding that travels carries four things, in this order, and each one answers a question the next reader will otherwise ask out loud.
- The question. What was being decided, in the words the business uses, which is never the statistical hypothesis or the platform's name for the experiment.
- The method. How the answer was produced, including the design, the population, the duration, the primary measure named in advance and the validity checks that passed.
- The number with its uncertainty. The effect in the units of the measure, the interval around it and the baseline the effect sits on.
- The decision it supports. What the analyst recommends and the threshold that recommendation rests on.
Max Shron makes a related argument about the front of the process in Thinking with Data, published in 2014, where he sets out context, need, vision and outcome as the four things a data project settles before anybody opens a query editor. The structure above is the same discipline applied at the other end, so that the write up answers the questions the scoping asked.
Recommending a decision is the part analysts leave out most often, usually out of a worry about overstepping. Leaving it out produces a decision made by somebody with less information, since the analyst is the person who knows the interval straddles the threshold.
One result written twice
The signup test from earlier in this module gives a concrete comparison. The weak version of the finding runs like this.
Tested the shorter signup form. The new version won, conversion is up 4.4 per cent, and shipping it is recommended.
Every sentence in it is true and the reader cannot act on it. There is no baseline, so 4.4 per cent could describe a rise from 45 per cent to 47 or from 0.4 to 0.42. There is no interval, so the reader cannot tell whether the gain is settled. There is no threshold, so the recommendation rests on nothing stated. The strong version carries the same result in four short paragraphs.
The question. Whether removing three fields from the signup form increases the number of people who complete it.
The method. A randomised test ran for three weeks with 120,000 visitors in each group. The primary measure was the completed signup rate, named before the test started. The split arrived within its expected ratio and no guardrail moved, so the result is valid.
The result. Completed signups rose from 4.50 per cent to 4.70 per cent, which is a relative gain of 4.4 per cent on a base of 4.50. The 95 per cent interval runs from a 0.7 per cent to an 8.2 per cent relative gain.
The decision. The practical threshold agreed before the test was a 2 per cent relative gain, below which the second form costs more to maintain than it returns. The interval straddles that threshold, so the recommendation is to run the test for two more weeks, which is expected to narrow the interval enough to settle the question either way.
The second version is four times as long and it is the only one of the two that a reader who was absent can act on. It also survives being forwarded, which the first version does not, since the first version depends entirely on a conversation the reader was part of.
Writing for the reader who was not in the room
Three details separate a finding that survives forwarding from one that does not.
The exact definition of the measure. Completed signup rate, counted per unique visitor within seven days of first arrival, is a definition. Conversion is a word that four people in the room define differently and never discover it.
The limits of what the result covers. A three week test on desktop traffic says nothing about the mobile application, and writing that down prevents the finding from being quoted next quarter as though it did.
The absolute numbers beside the relative ones. A relative gain with no baseline attached is the commonest way an analytics finding misleads a senior reader. A 40 per cent improvement on a measure sitting at 0.3 per cent sounds like a much larger result than it is.
The record that outlives the decision
A finding has one more job once the team has made its decision, which is to stop the same question being asked again in eighteen months by people who were not there.
That means the write up goes somewhere searchable, with the date, the measure, the effect, the interval and the decision, and stays there after the feature ships or is removed. A thread in a chat tool does not qualify, because nobody will find it. A slide deck does not qualify either, because the numbers inside it are pictures.
The value of the record compounds in a way a single finding does not. A team with two years of results can answer whether changes of this kind have ever worked before, which is a far better prior than anybody's intuition and the only defence against running the same test a third time.
One good finding, written well and filed where people can find it, changes one decision. Turning that into something an organisation relies on every quarter needs a set of habits around the tests, which is a different question from how any single test was run.