Data quality in analytics

Data quality in analytics is the question of whether a recorded figure corresponds to what actually happened inside the product.

The question sits underneath every method later in this course. Ron Kohavi, Diane Tang and Ya Xu give instrumentation a chapter of its own in Trustworthy Online Controlled Experiments, published in 2020, and the reason is structural. A funnel, a retention curve and a controlled experiment all assume the events arrived intact, and none of them has any way of noticing that they did not.

The difficulty for a product manager is that a broken number looks exactly like a business result. A figure that fell 30 per cent overnight has three possible explanations. A release broke the event, a bug broke the feature, or customers changed their behaviour. All three produce the same line on the same chart, and the only way to tell them apart is a check somebody set up in advance.

Collection fails quietly for a second reason as well. Nothing in the pipeline raises an error when an event stops arriving, because an event that is never sent produces no failure anywhere. The system is healthy, the service is up, the dashboards render and the data is wrong.

The sections below set out six ways collected data stops matching reality and the check that catches each. They then describe monitoring the collection itself, which is a different activity from monitoring the service. Finally they take reconciliation against a system of record and the honest way to report a loss nobody can recover.

Six ways collected data stops matching what happened

An event that never fires is the commonest and the most expensive. A release moves the export button into a new component and the tracking call is not carried across with it. Exports per week go from 4,100 to zero on the day of the release. No alarm sounds anywhere. Nobody opens that chart until the monthly review five weeks later, and those five weeks are gone permanently.

An event that fires twice is harder to spot, because a doubled figure still looks like a figure. It comes from a retry after a timeout, a form somebody submitted twice, or a component that mounts twice under a framework's development mode and ships that way. The symptom worth watching is a step that converts above 100 per cent, which is arithmetically impossible and therefore proof of a collection fault.

Collection lost in the browser removes events before they ever reach the pipeline. Ad and tracker blocking extensions stop analytics scripts loading alongside advertising, and GWI's figures for the second quarter of 2025, reported through DataReportal, put 29.5 per cent of internet users worldwide using an ad blocker at least sometimes. Events fired as a page is closing add a second source of loss, since a request started during unload is often abandoned. Neither loss is spread evenly, because blocker use is heaviest among technical people on desktop machines. The gap is biased.

Late arriving data makes a figure move after a team has read it. A mobile application queues events while the device is offline and sends the queue when the connection returns, so Monday's numbers keep growing on Tuesday and Wednesday. One product read Monday's active count as 18,400 on Tuesday morning, 19,100 on Wednesday and 19,350 by Friday, which is 5.2 per cent above the first reading. A team comparing Monday against the previous Monday at the same hour is comparing a settled figure with an unsettled one. The comparison is not fair.

Schema drift is a property changing shape without anybody declaring it. A value that was always a number starts arriving quoted as text from one client version, and the rows either fail to parse or land somewhere nobody is looking. An enumerated property gains a sixth allowed value in a release, and a report filtering on the original five silently excludes everybody on the new one. Nothing warns the reader.

A renamed field is schema drift with a release date attached. A team renames plan_type to subscription_tier and both names now exist, each holding part of the history. The version of that story worth remembering involves two platforms. The iOS application shipped the new name in week one and the Android application shipped it in week four. For three weeks the two platforms reported the same fact under two names, and every chart split by plan showed a gap that looked like a change in behaviour.

Monitoring the collection itself

A team notices these failures by watching the data rather than by trusting it, and five checks cover most of the ground.

1. Volume per event against the same weekday a week earlier. Traffic has a weekly shape, so comparing Tuesday against Monday produces false alarms every week and comparing Tuesday against last Tuesday does not.
2. An alert on an event that has stopped entirely. This is the cheapest check to build and it catches the most expensive failure on the list.
3. A null rate per required property. A property the tracking plan calls required, arriving empty on nine per cent of rows, is a bug somebody shipped and nobody reported.
4. Schema validation at the edge of the pipeline. An event whose properties do not match the declared types is rejected or quarantined at the moment it arrives, with a count somebody can see.
5. A settling curve for late data. Measuring how long a day takes to finish arriving lets a team state a cut off, so a report covering the week to Monday is read on Friday, once the week has finished landing.

Watching the data is a different activity from watching the service, and teams that have the second often assume they have the first. A service can have perfect uptime, no errors in its logs and healthy response times while emitting the wrong events or no events at all. The service monitors whether requests succeeded. Data monitoring asks whether the things that happened were recorded correctly, and nothing in the first answers the second.

Every check also needs a named owner and an alert that reaches a person. Monitoring routed to a dashboard becomes one more dashboard nobody opens, which is the failure this whole section exists to prevent.

Reconciling against a system of record

The strongest check available to a product manager needs no engineering work at all, which is comparing an analytics figure against a system that has an independent reason to be correct.

Billing records every payment because a company would notice money going missing. The mail provider records every delivery because customers complain when a message fails. The support system records every ticket because somebody has to answer it. Each of those is a system of record, and each of them has an analytics event that should match.

The comparison is blunt and effective. One product found 12,400 Payment Recorded events in its analytics tool against 11,050 payments in the billing records for the same month, which is 12.2 per cent more events than payments. Payments do not appear from nowhere, so the excess was a collection fault, and the cause was a retry that fired the event again when the first confirmation timed out.

A standing monthly reconciliation of two or three figures catches most of what the automated checks miss. It takes twenty minutes, it needs no access beyond a report and a billing export, and it is the single highest value habit in this module.

The loss a team cannot recover and should state

Some collection is gone and no amount of monitoring brings it back, which is worth saying plainly before a team starts apologising for it.

An event that was never implemented in March cannot be recovered in June, because nothing recorded the action at the time. A script a blocker prevented from loading wrote nothing anywhere. A Safari visitor whose identifier expired after seven days cannot be joined to their earlier history. Those losses are permanent. A team still hunting for a way to recover them is spending time it could spend on the next measure.

The useful response is to measure the size of the loss and report it beside the figure. A different product, reconciling against its own billing records, found that analytics captured 89 per cent of payments and that the shortfall sat mostly on desktop. That sentence turns an unknown into a known quantity, and it stops the argument about whether the number is real.

A stable undercount also behaves better than teams expect. A figure capturing 89 per cent of reality every month still moves correctly when the product changes, because the missing eleven per cent moves with it. What breaks that is a change in the share itself, which happens when a browser ships a new restriction or a blocker list adds the analytics vendor. Running the reconciliation monthly is how a team finds out that the share moved, and comparing two segments with different capture rates is the one thing a stable undercount never makes safe.

Collection a team can defend is what makes the rest of the subject worth doing. With events arriving, properties intact and a known capture rate written down, the behaviour in the data becomes readable. The first thing worth reading is a sequence of steps and the point at which people stop, which is where the next module begins.