Segmentation in analytics is the practice of cutting a measured population into groups that behave differently, so that a single average stops standing in for people who have nothing in common.
Every figure in this course so far has been an average over a mixed population. A funnel conversion of 1.2 per cent, a retention floor of 22 per cent and a mean of six active days each describe a crowd, and each of them is the answer to a question nobody asked in that form.
The vocabulary came from marketing, where a segment is a group of buyers a company chooses to serve. Analytics borrowed the word and lowered the stakes, because a segment in a chart is a filter on a query and commits nobody to anything. That difference matters, since an analytics segment can be created in ten seconds and a marketing segment takes a quarter to agree.
A product manager needs this on the day a headline number moves and nobody can say why. Somebody will ask what changed, and the only useful reply is a cut of the same number that shows which part of the population moved. A team without that cut is left reasoning about a crowd.
The sections below give the cuts that usually earn their place. Simpson's paradox is then set out with a worked case where every segment improves and the total falls, followed by the arithmetic that separates the two effects. Finally they cover the discipline that keeps this from becoming a search for a flattering number.
Cuts that separate people who behave differently
Four cuts do most of the work across most products, and they share one property worth naming. Each of them describes a difference that existed before the person arrived, which is what makes the groups comparable.
- How somebody arrived. Paid search, an invitation from a colleague, an app store listing and a partner integration produce populations with different intent, and intent moves every measure downstream of it.
- What they pay. Free, trial, self serve paid and enterprise are four different products from the customer's side, whatever the code says.
- What device they are on. Screen size changes what is possible, and it also stands in for context, since a phone at a bus stop and a laptop at a desk are different working conditions.
- How long they have been here. A customer of two years and a customer of two days meet different products, because one of them has already set everything up.
A fifth cut is tempting and needs care. Splitting by how much somebody uses the product puts heavy users in one group and light users in another, which guarantees a difference in every measure of use and explains nothing. That cut answers questions about what heavy users do next, and it never answers questions about what made anybody heavy.
Simpson's paradox and where the name came from
Choosing cuts carefully still leaves one surprise, which is a set of segments that all move one way while the total moves the other. Edward Simpson described the effect in the Journal of the Royal Statistical Society in 1951, showing that an association present in every subgroup of a table can vanish or reverse when the subgroups are added together. Colin Blyth attached Simpson's name to it in 1972, and the label stuck.
The case that made it famous concerns graduate admissions at the University of California, Berkeley, in the autumn of 1973. Bickel, Hammel and O'Connell reported in Science in 1975 on the 12,763 people who applied that year. About 44 per cent of the 8,442 men were admitted against about 35 per cent of the 4,321 women, while most individual departments admitted women at a rate equal to or higher than men. The explanation was that women applied more often to the departments admitting a small share of everybody.
Nothing about that example is a statistical curiosity. It is the ordinary consequence of combining groups of unequal size with unequal rates, and it happens in product data every time an acquisition mix moves.
One rate improving in every segment while the total falls
The worked example is the signup rate at a booking product, compared between March and April. In April the team shipped a redesigned form, and in the same month marketing ran a campaign on a mobile advertising network.
| Device | March visits | March signups | March rate | April visits | April signups | April rate |
|---|---|---|---|---|---|---|
| Desktop | 3,000 | 600 | 20.0 per cent | 1,500 | 330 | 22.0 per cent |
| Mobile | 2,000 | 140 | 7.0 per cent | 3,500 | 315 | 9.0 per cent |
| Combined | 5,000 | 740 | 14.8 per cent | 5,000 | 645 | 12.9 per cent |
Every figure in that table is arithmetic anybody can repeat. Desktop improved from 600 out of 3,000 to 330 out of 1,500, which is 20.0 per cent rising to 22.0 per cent. Mobile improved from 140 out of 2,000 to 315 out of 3,500, which is 7.0 per cent rising to 9.0 per cent. Total visits were 5,000 in both months. Total signups fell from 740 to 645, so the combined rate fell from 14.8 per cent to 12.9 per cent.
The cause is the mix. Mobile carried 40 per cent of March traffic and 70 per cent of April traffic, and mobile converts at well under half the desktop rate in both months. Moving weight from the strong segment to the weak one pulls the combined figure down further than either segment's improvement pushes it up.
A team reading only the combined row concludes that the redesigned form made things worse and reverts it. A team reading the segments concludes that the form is better on both devices and that the campaign bought traffic which converts poorly. Those two conclusions lead to opposite work, and the second one is supported by the same numbers the first one used.
Holding the mix constant to see the change underneath
Separating the two effects takes one more calculation. Apply April's rates to March's traffic mix, which means 3,000 desktop visits at 22.0 per cent and 2,000 mobile visits at 9.0 per cent. That produces 660 plus 180, or 840 signups from 5,000 visits, which is 16.8 per cent.
So the redesigned form is worth roughly two points of signup rate, taking the mix adjusted figure from 14.8 per cent to 16.8 per cent. The campaign is worth minus 3.9 points, taking the mix adjusted 16.8 per cent down to the 12.9 per cent actually observed. The two effects add to the net fall of 1.9 points the combined row reports.
Both findings are real and they point at different people. The product team shipped an improvement. The marketing team bought traffic that converts at 9 per cent, and whether that was a good purchase depends on what a mobile signup is worth compared with a desktop one, which is a question about revenue and no longer about conversion.
The discipline that keeps segmentation honest
Segments multiply fast. Four cuts with four values each produce two hundred and fifty six combinations, and a team that looks at all of them will find something that looks meaningful in a population of eight people. The defence is to decide the cuts that matter before looking at the data, and to write them down where somebody else can see the list.
Two further habits keep the practice usable. Any segment reported on a dashboard carries its own denominator next to the rate, so a reader can see that 40 per cent means four people out of ten. And a segment small enough to swing on individual behaviour gets reported as a count, since a percentage implies a precision the numbers do not have.
There is a deeper limit under all of this, and the worked example shows it. The mix adjusted figure of 16.8 per cent was built by assuming that April's desktop visitors would have behaved the same way if the campaign had never run. That assumption is reasonable here and it is still an assumption, and observational data offers no way to check it.
Reading behaviour reaches its limit at this point, because the question a team actually has sits beyond it. Every pattern found so far describes people who differed for their own reasons, and deciding what a change would do requires making the change to a group chosen at random and leaving a comparable group alone.