The vocabulary the exams and the daily work both assume you already have. Every entry gives the definition, the source it is drawn from, and the misreadings that follow from getting it wrong.
39 concepts in Metrics, page 2 of 2
Why examining many measures at the five per cent threshold produces winners out of nothing, the arithmetic for twenty measures and for ninety six segment cuts, the two corrections available and the practice that avoids needing either.
Metrics
The effect that fades once a change stops being new, the effect that grows once people have learned their way around it, the two readings that separate them from noise and the published evidence on how often either one decides anything.
Metrics
Output is what the team produced. Outcome is what changed for someone because of it. A product owner is accountable for the second and is usually measured on the first.
Metrics
Which facts inside an event property count as personal data, the lawful basis and the four tests a consent has to pass, how long a record may be kept, and the arithmetic showing what a declined consent rate does to every funnel and cohort the product reports.
Metrics
What product analytics is, how it differs from business intelligence and from user research, the event as the unit everything else is counted from, and the single test that separates a practice from a wall of dashboards.
Metrics
Use product economics to compare investments, calculate customer contribution and avoid misleading lifetime value or prioritisation claims.
Metrics
AARRR from Dave McClure in 2007 and HEART from Google's research team in 2010, what each one was built to solve, the Goals Signals Metrics process that sits underneath HEART, and the honest criticism of both alongside the north star.
Metrics
The four kinds of number a product produces, what each one is good for and how each one misleads, a worked case where the mean and the median disagree by the width of the whole population, and the property that separates a measure a team can act on from one that only ever rises.
Metrics
Watching how the product performs in the market, what it costs the company to run, and what it contributes to profit, after the launch has stopped being news.
Metrics
How a measured difference becomes a decision to ship, iterate or abandon, what the flat result actually says, the eight result shapes and the decision each one supports, and why shipping a change that did nothing has a cost.
Metrics
The shape a cohort traces over time, what a curve that flattens says about a durable audience, what a curve that reaches zero says about the absence of one, and how far apart day N, unbounded and bracket retention put the same cohort on the same data.
Metrics
The validity check that runs before any result is interpreted, the chi squared arithmetic behind it, the five stages of an experiment where a mismatch is created and why a mismatched test is discarded.
Metrics
The minimum detectable effect, statistical power and the relationship that ties them to the number of people a test needs, a table running from a one per cent improvement to a fifty per cent one with the arithmetic shown, and what follows honestly for a product with modest traffic.
Metrics
Cutting a population so an average stops describing nobody, the cuts that usually pay, and a worked case of Simpson's paradox where a signup rate improves on every device and the combined figure falls from 14.8 per cent to 12.9 per cent.
Metrics
What a p value claims and what it cannot claim, what a confidence interval adds that a verdict of significant throws away, and why a difference can be statistically significant and still too small to fund.
Metrics
One measure of the value a product delivers, the input measures beneath it that a team can move directly, the four properties a candidate has to pass before anybody commits to it, and the parts of a business a single number will never describe.
Metrics
What repeated checking does to the false positive rate of a running test, the published figures for one check against ten, why the habit is invisible from inside a team and the two legitimate ways to stop a test early.
Metrics
The document that holds every event a product records, the columns each row carries, the review a proposed event goes through, how a plan is versioned and how an event is retired, and why a plan that exists only inside the analytics tool has no owner at all.
Metrics
The limit behavioural data reaches, which is that it records what happened and never why, what session replay, surveys and interviews each recover from that limit and how each of the three misleads, the population no product measure contains, and the decisions no number settles.
Metrics