Product analytics is the use of behavioural data from a product itself to decide what a team builds next.
The practice grew out of web analytics. When the Web Analytics Association published its standard definitions in 2008, the units at the centre of it were the visit and the page view, because a website was the thing being measured. A product is a sequence of actions somebody performs, so the unit had to move. Product analytics counts events, meaning a record that a particular person did a particular thing at a particular moment, and every event carries properties that describe the particular case.
That change of unit is what brings the subject onto a product manager's desk. Nobody outside the team knows which action in a sign up flow counts as finishing it, which screen the team meant people to reach, or which of forty buttons is worth recording at all. The definitions come from the person who decided what the product should do, and a product manager holds more of those definitions than anybody else in the organisation.
An analytics practice fails quietly. A team buys a tool, spends a fortnight installing it and within a quarter has eleven dashboards that nobody opens between planning meetings. Every number on them is correct. None of them has ever caused anybody to stop building something, which is the only outcome that would have paid for the tool.
The sections below separate product analytics from the two other ways an organisation learns about its product, which are business intelligence and user research. They then describe the event, which is the unit every later measure is built from. Finally they give the test that decides whether an analytics practice has earned its cost.
Three kinds of evidence about a product
Business intelligence and user research answer questions about the same product that product analytics does. Each of the three asks something of its own and produces its own kind of evidence in reply.
| Practice | The question it answers | The evidence it produces |
|---|---|---|
| Product analytics | What did people do inside the product | Events the product recorded as people used it |
| Business intelligence | How is the company performing | Transactions, invoices, contracts, renewals and support tickets |
| User research | Why did people do it | What people said, and what a researcher watched them do |
Product analytics and business intelligence are the two a team runs together most often, because both produce numbers and both arrive in a chart. The difference is what each one counts. Business intelligence counts what the company recorded about a customer, so it reports revenue, contract value and ticket volume, and it answers to finance. Product analytics counts what a person did inside the product, so it reports the actions themselves, and it answers to the team building the thing.
One example shows what each is worth on its own. Business intelligence reports that revenue from new accounts fell four per cent in March. Product analytics reports that the share of new accounts creating a second project within fourteen days fell from 38 per cent to 29 per cent in the same month, and that the fall started on the day of the March release. The revenue figure tells a board the size of the problem. The share creating a second project tells the team where to look.
User research sits further away from both, because it produces evidence of a kind neither of the others can generate. Product analytics can report that 61 per cent of the people who opened the invite screen closed it without sending an invitation. Only research can find that eight of them expected the invitation to carry a personal message and could not find a field for one. Behavioural data is a reliable record of what happened and it is silent about why, so a team that only counts events ends up inventing motives for its own users.
The event as the unit of measurement
An event is a record that somebody did something at a particular moment. A single row holds the name of the action, the time it happened, an identifier for the person who did it and a set of properties describing the particular case.
An invoicing product might record an event called Invoice Sent with four
properties, which are the account it was sent from, the channel it went out
on, the number of line items on it and the invoice total. Those four
properties are what turn one row into a hundred possible questions. Invoices
sent per account per month, the share sent by email, the effect of line item
count on how quickly an invoice is paid, and the total value sent by accounts
in their first thirty days all come out of the same recorded action.
Every measure later in this course is arithmetic over events. A funnel counts the people who produced one event and then another. Retention counts the people who produced any event in a later week. An experiment compares the rate of one event between two groups. Recording events is cheap, and deciding which actions deserve to be events is where the judgement sits.
The test of an analytics practice
A practice earns its cost when a number changes a decision, and that is the only test worth applying to it.
A team that can name three decisions from the last two quarters that went differently because of a number has a practice. One team stopped building a bulk import feature after finding that the accounts asking for it produced fewer than four imports a month between them. Another team moved a pricing page change to the top of its backlog after finding that people who reached the page from the in product prompt converted at three times the rate of people who reached it from search. Both are small decisions and both were changed by evidence.
Alistair Croll and Benjamin Yoskovitz put the same test at the centre of Lean Analytics in 2013, where the property that mattered most in a metric was whether it changed how the team behaved. A number that would not have moved a decision either way is costing instrumentation time, storage and attention, and returning a feeling of rigour.
What behavioural data cannot answer
Behavioural data has a boundary, and knowing where it falls prevents most of the arguments a team has about it.
Behavioural data cannot report on anything the product does not observe. A person who gave up before reaching the product, a decision made in a procurement meeting and a competitor's release are all invisible to it. It cannot report a motive, which is what the invite screen above shows. It cannot describe a population the product never reached, so it is useless for sizing a new market. And it answers slowly on anything that happens rarely, because a product with two hundred enterprise accounts produces too few renewals in a quarter to read a trend from.
A decision needs a number behind it. The numbers a product emits are not equally able to carry one, since some of them rise every month whatever the team does and some describe a population so mixed that no single figure belongs to anybody in it. Separating a usable measure from a flattering one is where the next page starts.