An LLM application needs defined outcomes for requests it cannot answer successfully. Missing evidence, ambiguous intent, a policy refusal and a technical failure are different conditions. Each needs an appropriate response and a separate measurement.
A single fallback such as “Something went wrong” conceals the cause. A generic retry can make the situation worse by turning a legitimate refusal into repeated attempts to obtain the same disallowed result.
Separate model outcomes from service failures
Use application statuses that describe what happened:
| Outcome | Meaning | Typical next step |
|---|---|---|
| Answered | Sufficient evidence supports a response | Validate and display the answer |
| Insufficient evidence | Necessary information is absent | Explain the gap or retrieve additional authorised evidence |
| Ambiguous | Different interpretations require different answers | Ask one focused clarifying question |
| Declined | The request is outside permitted scope | Explain the relevant boundary or offer a supported route |
| Incomplete | Generation stopped before completion | Apply a bounded recovery policy |
| Service error | The provider or a dependency failed | Retry when appropriate or report temporary unavailability |
Provider refusal signals may arrive outside the application's requested schema. Normalise those signals into the application contract without pretending that the model returned a valid answer object.
Give missing evidence a valid representation
A response schema should allow an outcome other than answered. An example application object is:
{
"outcome": "insufficient_evidence",
"answer": null,
"citations": [],
"missing": "The supplied policy does not cover international returns.",
"clarifying_question": null
}
The application must check relationships between fields. An answered response may require supporting citations; an ambiguous response may require a question. A flat schema with nullable fields can be portable, while a discriminated union can express the contract more tightly where supported.
Standard JSON Schema offers schema combination keywords, but provider support varies. When using closed objects in separate branches, declare allowed properties in the relevant branch. A top level additionalProperties restriction can otherwise reject properties declared only inside a branch.
Ask for clarification when the distinction matters
A request such as “Cancel it” is ambiguous if the conversation contains two active orders. Ask which order the customer means before proposing an action.
Do not ask a question merely because the wording permits a harmless variation. Clarification is useful when the missing detail changes the decision, the data needed or the consequences.
Keep the question specific. “Do you mean order A-4471 or A-4472?” is easier to answer than a general request for more context.
Validate citations before showing an answer
Check that every cited identifier belongs to the evidence supplied for the request. Unknown IDs indicate a contract failure; they do not reveal exactly how the model produced the error.
Depending on the product, a bounded regeneration, renewed retrieval or human handoff may be appropriate. Replacing the ID with a valid one without rechecking the claim can create a misleading citation.
Then assess whether the evidence supports the claim. Identifier validity alone is insufficient.
Make refusal and abstention measurable
Track appropriate abstention on unanswerable cases and unnecessary abstention on answerable cases. A system that refuses every request has avoided unsupported answers but has failed its purpose.
A rise in insufficient evidence responses may reflect an index problem, new user needs or a model change. Inspect cases before assigning a cause. Similarly, a zero rate is a reason to audit known source gaps, not proof of either perfect coverage or universal failure.
For a practical test, include one supported question, one absent fact, one ambiguous action request, one refusal triggering request and one simulated timeout. Each should reach a different, deliberate outcome.