A product metric is a number worked out from recorded behaviour and used to judge whether a product is doing what it was built to do.
Every product metric is arithmetic over the events a product emits, which means one week of collection can produce several hundred of them. Very few earn a place in a weekly review. What separates the few from the many is partly the kind of number each one is, and partly whether anybody in the room can say what they would do if it moved.
The kind matters because each kind fails in its own way. A count rises with the size of the population. A rate depends entirely on a denominator that usually goes unstated. A ratio moves when either of its two halves moves. An average hides the shape of the population it was taken over, and the hiding is worst exactly where product data lives, which is a long tailed distribution with a small group of heavy users at one end.
The sections below set out the four kinds and what each one is good for. They then work one case through with real arithmetic, where the mean and the median of the same thousand accounts disagree completely. Finally they give the property that decides whether a measure belongs in a review at all.
The four kinds of number a product produces
| Kind | What it is good for | How it misleads |
|---|---|---|
| Count | Size, load and capacity planning | It rises with the population, so growth flatters it |
| Rate | Behaviour per person or per period | The denominator decides the answer and is often unstated |
| Ratio | Comparing populations of different sizes | Either half can move, so the cause is ambiguous |
| Average | One figure standing for many people | It conceals the distribution and follows the heaviest users |
A count is a total over a stated period. Invoices sent in September, accounts created last week, support tickets opened since the release. Counts answer questions about size and load, so they belong in capacity planning and in any conversation about cost. The failure is that a count rises as the population rises, which means a growing product improves its counts without improving anything anybody did.
The cumulative count is the worst member of the family, because a running total since launch can only go up. Registered accounts since launch rose by 400 in a month when 380 existing accounts stopped opening the product entirely, and the figure recorded none of that.
A rate is a count divided by a period of time or by a population. Invoices per active account per month, errors per thousand requests, sessions per person per week. A rate removes the growth of the population from the answer, which is what makes it readable across a year. The failure is that the denominator carries most of the meaning and is usually left out of the sentence. Sessions per user per week means something different for every user a team might have counted, and a definition of active that includes anybody who ever registered gives a different number from one that includes anybody who opened the product this month.
A ratio divides one count by another where both describe the same population. Conversion from one step to the next, the share of accounts that invited a colleague, the proportion of repairs closed on a first visit. A ratio is bounded and it compares across populations of unlike size, which is why most of the later measures in this course are ratios. The failure is that two numbers move independently, so a rise in conversion can mean more people finished or fewer people started.
An average is a single figure standing for a whole population. The mean adds everything up and divides by the count of people. The median is the figure in the middle when everybody is lined up in order. Both are averages and they answer different questions, which the next section works through.
What an average hides about its own distribution
An invoicing product counted how many invoices each active account sent in September. A thousand accounts were active. Between them they sent 1,290 invoices, so the mean is 1.29 invoices per account. The distribution behind that mean is set out below.
| Invoices sent in September | Accounts | Invoices |
|---|---|---|
| None | 620 | 0 |
| One | 210 | 210 |
| Two | 90 | 180 |
| Three | 60 | 180 |
| Eight | 15 | 120 |
| One hundred and twenty | 5 | 600 |
| Total | 1,000 | 1,290 |
Lined up in order of invoices sent, the five hundredth account of the thousand sent none, so the median is zero. The mean is 1.29 and the median is 0. Both are correct arithmetic over the same thousand accounts, and a team quoting only the mean has described a population accurately and told nobody anything about it.
Two facts the mean concealed matter more than the mean itself. Almost two thirds of the accounts counted as active sent no invoice at all during the month, which makes the word active a claim worth checking before anything is built on it. And five accounts, half of one per cent of the population, sent 600 of the 1,290 invoices, which is just under half of everything the product carried that month.
A roadmap built for the account sending 1.29 invoices a month is built for an account that does not exist. The 620 accounts sending none have an onboarding problem the product has not solved. The five accounts sending 120 need bulk tools, an import route and somebody to ring when a run fails. No single change designed around the mean helps either group.
The mean earns its place when a distribution is packed closely around it, as the length of a checkout flow in seconds usually is. For almost everything a product counts per person, the distribution has a long tail at the high end, and the useful summary is the median together with two or three percentiles. Reporting the median, the 75th percentile and the 95th percentile of invoices per account would have shown the shape above in three numbers.
The difference between an actionable measure and a vanity metric
A measure is actionable when somebody in the room can name the work that would move it. A measure is flattering when it rises whatever anybody does, so the chart looks encouraging every month and carries no information about the month.
Eric Ries drew that line in The Lean Startup in 2011, naming the two kinds a vanity metric and an actionable metric, and gave an actionable one three properties. A movement in it points at something somebody decided to do. The people expected to act on it can follow the arithmetic behind it. And anybody who doubts the figure can go back to the records underneath and check it for themselves.
Alistair Croll and Benjamin Yoskovitz set out four properties of a good metric in Lean Analytics in 2013.
- It is comparative, so the figure can be set beside the same figure last month or beside the figure for another segment.
- It is understandable, so people can hold it in their heads and argue about it without opening a spreadsheet.
- It is a ratio or a rate, which moves for a reason somebody can name.
- It changes how the team behaves, which the two authors called the most important property of the four.
The fourth property is the one that fails most often in a real review. A team can hold a measure that is comparative, understandable and expressed as a rate with no answer at all to the question of what it would do differently if the figure fell by a fifth. A measure nobody would act on is a measure nobody needs to collect.
Four properties keep a poor measure out of a review. They say nothing about which of several good measures a team should steer by, and a team carrying six defensible measures pulls in six directions on the first week that two of them disagree. Choosing one measure to sit above the others is where the next page goes.