Measuring an AI feature

Measuring an AI feature is the work of judging a generative capability already in front of customers, using measures that survive the fact that it answers the same question differently on two consecutive attempts.

Every measure in this course so far rests on an assumption nobody states. A funnel step, a retention curve, an activation definition and a segment all assume that two people who took the same action met the same product. Export to CSV produces a file. The file is correct or the export failed, and the event says which.

A generative feature holds none of that fixed. The same prompt produces different wording each time, the output is better or worse by degrees with no line separating success from failure, and the model behind it can change under the team when a provider updates what a version alias points at. Three properties that every earlier measure treated as fixed are all moving at once.

The distinction worth drawing at the start is between a model and a feature. Evaluating a model asks whether a set of weights answers a test set correctly, and a research team does that work on public benchmarks and private sets before anything ships. Measuring a feature asks whether the thing customers are already using does useful work for them, on their own inputs, at the speed and the cost the product pays. A model can score well on both counts and still ship inside a feature nobody accepts.

The work lands on a product manager because nothing in the pipeline reports a wrong answer. A failed payment produces an error. A hallucinated citation produces a normal looking event, a normal latency and a satisfied service monitor, and the only record that anything went wrong is what the customer did next.

The sections below take acceptance rate and the unit it is counted in, then edit and override rate, then the error rate a team builds for itself. The trade between latency and quality follows with arithmetic. The page then sets out why the ordinary engagement measures point the wrong way here, and the properties every event from the feature carries.

Acceptance rate and the unit it is counted in

Acceptance rate is the share of the things the feature offered that somebody took. It is the first measure worth building for any feature that suggests, drafts or completes, because acceptance is an action the customer performs and the product can record it without asking anybody a question.

Building it needs two events and the first one is the one teams forget. An accepted event on its own gives a count with no denominator. A shown event, fired at the moment the suggestion reached the screen, is what turns that count into a rate, and it has to fire on every suggestion including the ones nobody looked at.

The unit those two events count is part of the definition and changes the answer. Gal Bakal, Ali Dasdan, Yaniv Katz, Michael Kaufman and Guy Levin published a report on 23 January 2025 covering more than 400 developers at ZoomInfo using GitHub Copilot. They reported an average acceptance rate of 33 per cent for suggestions and 20 per cent for lines of code. One feature, one population, one period, and two figures thirteen points apart, because a suggestion counts once when the unit is a suggestion and once for every line it holds when the unit is a line.

Four units come up and each answers a different question.

1. Per suggestion shown. How often the feature guessed well enough to be taken. The usual headline.
2. Per request the customer made. How often somebody who asked for help got something they used. The honest denominator, since it includes the attempts that produced nothing.
3. Per line or per character. How much of the work the feature did. The figure that connects to any claim about time saved.
4. Per hour of work. How much the feature contributed over a working day. The figure a finance conversation eventually asks for.

Acceptance rate on its own rewards caution. A feature tuned to suggest only when it is confident offers fewer suggestions, gets a higher share of them accepted and does less work for the customer, so the rate rises while the product gets worse. Pairing it with accepted suggestions per hour closes that gap, because the pair moves together only when the feature is offering more and being right about it.

Edit rate and override rate

Acceptance records a decision taken in one second. What happened to the output afterwards is a separate measure and it usually tells a different story.

Edit rate is the share of an accepted output that somebody changed, measured a fixed interval after acceptance. A drafted email kept word for word and a drafted email rewritten from the second sentence onward are both acceptances, and only the edit rate separates them. Override rate is the same idea for a feature that acts without asking, meaning the share of the feature's decisions a person reversed. A ticket the model routed to billing and a human moved to technical support is an override, and the product already records the move.

Override rate is the cheapest quality signal available to most teams, because the reversal is a deliberate act somebody takes inside the product for their own reasons. It needs no survey and no sampling. The limit is that it exists only where the feature decides something a person can undo.

The relationship between acceptance and survival produced a result worth knowing. Albert Ziegler and colleagues at GitHub published Productivity Assessment of Neural Code Completion in May 2022, matching what developers said about their own productivity against the usage measurements recorded for those same developers. They found that the rate at which shown suggestions were accepted predicted developers' own sense of their productivity better than any measure of how long the completions persisted in the code afterwards. The authors had expected the opposite, since persistence looks like the more careful measure.

Two readings of that finding are useful and they do not compete. Acceptance tracks what the feature felt like to use, which is what decides whether anybody keeps using it. Persistence tracks what the feature actually delivered, which is what decides whether the company got anything for the money. A product with high acceptance and heavy editing is a fast drafting tool, and that is a legitimate product so long as the team knows which one it shipped and stops claiming the other one.

Error rate and hallucination rate

Edit rate says an output changed and never says it was wrong. A confident, fluent, entirely fabricated answer that nobody checked is accepted, unedited and counted as a success by every measure above.

So the error rate is the one measure a team builds rather than collects. The method is a graded sample. Each week somebody draws a fixed number of real outputs. A colleague who took no part in writing the prompt then grades each one against a written standard. The grades go into a table carrying the date, the model version and the rate, and that table is the only record the product holds of whether the feature is right.

Grading needs categories rather than a pass mark, because the failures differ in what they cost.

1. Correct and complete. The answer is right and covers the question.
2. Correct and partial. Nothing in it is wrong and something is missing.
3. Wrong. A claim in it is false.
4. Fabricated source. A citation, a figure or a quotation that does not exist, which is the failure worth counting separately because a reader checking the answer will trust the citation and stop.
5. Refused. The feature declined a request it should have handled, which costs the customer a whole interaction and appears nowhere else.

The size of the rate teams find when somebody looks properly is the argument for doing it. Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher Manning and Daniel Ho published the first preregistered evaluation of commercial legal research assistants as a preprint on 30 May 2024, and afterwards in the Journal of Empirical Legal Studies. They reported that the tools sold by LexisNexis and Thomson Reuters produced hallucinations between 17 and 33 per cent of the time, against published claims from those vendors that the technique behind the products removed hallucinations.

Two things about that result transfer to any team. The rate was large enough to change what a professional should do with the tool, which makes it a product finding as much as a research one. And it took an outside group with a graded question set to find it, because nothing in the products' own usage data distinguishes a fabricated citation from a real one.

Sample size decides how much a weekly rate can say. A hundred graded outputs put an error rate near 20 per cent inside an interval of roughly eight percentage points either way, which is enough to notice a rate that has doubled and nowhere near enough to notice one that moved by a tenth. A team that wants the smaller change has to grade more, and the honest position is to publish the interval beside the rate so nobody reads noise as a trend.

The trade between latency and quality

A graded rate improves with almost any change that lets the model do more work, and every one of those changes is paid for in the customer's waiting time. A larger model, a second pass over the draft, a retrieval step and a self check all raise quality and add seconds.

One drafting feature shows what the choice looks like when the arithmetic is written out. Two models are on offer and the team measured a fortnight of real traffic against each.

The two rows in the middle are where the argument usually goes wrong. Taken on its own, acceptance says the slow model is better by 13 points, which is about 42 per cent in relative terms. Counted per request, it is better by three drafts in a hundred, which is about 10 per cent, because more than a quarter of the people who asked it for something gave up before it answered. The headline gap is four times the real one.

The last row moves the answer again. Once the light editing filter is applied, the slow model produces 22.8 usable drafts per hundred requests against 11.1, which is roughly twice as many. Three measures over the same fortnight of traffic say the slow model is 42 per cent better, 10 per cent better and 105 per cent better, and each of the three is arithmetically correct.

The denominator rule this course applied to funnels settles which figure to use. The right denominator is whatever the customer asked for, so the request count is the right base and the count of suggestions shown is the wrong one, because the abandoned requests are exactly the damage the latency did. A rate computed over the people who waited is a rate computed over a group the slow model selected for patience.

One effect sits outside the table and belongs in the decision anyway. A feature that takes six seconds gets asked fewer questions in the following weeks, because people stop reaching for it, and that loss shows up in the request count over a month. Measuring the two models on a single fortnight holds the request volume roughly fixed and therefore prices the latency too cheaply.

Why ordinary engagement measures mislead for a generative feature

Latency is at least a measure with an agreed direction. Most of the engagement measures earlier in this course lose their direction entirely once the feature is generative, because the thing they count rises when the product fails.

Every row in that table shares one property. The measure counts effort, and effort is what a customer spends when the feature works badly, so the chart climbs on a bad week and a team reading it alone will celebrate.

Two further problems have no equivalent anywhere else in this course. Segmentation cannot repair the variation, because the variation sits inside the feature and never across the population. Two customers in the same segment sending the same prompt received different answers, so cutting the data finer narrows the population and leaves the spread where it was. What answers a question about spread here is a distribution of graded outputs and never a finer average.

A model change is the other problem without a parallel elsewhere. A provider updating what a version alias points at changes the feature for every customer at once, with nothing in the team's own repository to mark the day. A chart that steps on a Tuesday with no deploy behind it is the signature, and confirming it takes the model version recorded on every event.

The properties every event from the feature carries

Confirming that a step in a chart came from a model change means the event already carried the version, which makes the tracking plan for a generative feature longer than the plan for anything else in the product.

1. Model name and version, as the provider reports it back. The version the code requested and the version that answered can differ.
2. Prompt template version. A prompt edited on a Thursday is a product change and needs a number.
3. Retrieval index version and the documents returned. A feature grounded in a changing corpus changes when the corpus does.
4. Sampling settings. Two requests with different temperature settings are two different products.
5. Latency and token counts. Latency for the trade above, tokens because they are what the feature costs.
6. The assignment, if a test is running. Which version of the feature this person was served.

Token counts earn their place for a reason no other measure covers. A generative feature has a marginal cost per use that scales with how much the customer asked for, which almost nothing else in a software product does, so cost per accepted output belongs to the product team as much as to finance. A change that raises acceptance from 31 to 34 per cent and triples the tokens spent getting there is a decision somebody should make deliberately.

With all six properties on the event, a team can say what the feature did last Tuesday and still cannot say what caused a change in it. Acceptance rising three points after a prompt edit has at least four explanations, which are the edit itself, a silent provider update, a different mix of people arriving that week and the ordinary variation of a system that answers differently every time. Separating them means serving the old version to a randomly chosen half of the traffic and comparing, which is the instrument the next module of this course is built on.