This capstone combines the course's prompt, retrieval and agent exercises into a reviewable application design. The deliverable is an order support prototype and a release packet showing what it can do, how it fails and which evidence supports the release decision.
A production deployment is not required. Use fictional records and simulated side effects. The project should be complete enough that another practitioner can reproduce the tests and understand its limits.
Define a narrow product contract
The assistant answers cancellation questions from a versioned policy, retrieves the current order state and proposes a cancellation only when the customer explicitly requests it.
An informational question must not become an action. “Can I cancel?” and “Please cancel this order” need distinct treatment.
Before implementing the workflow, define the evidence and response fields required for each allowed outcome. Include answered, clarification_required, insufficient_evidence, declined and action_completed so a refusal or missing fact cannot be mistaken for a completed action.
Assemble the prototype from the labs
Use the prompt evaluation lab to establish a baseline and compare instructions. Adapt the classification task to separate informational questions from action requests.
Use the retrieval lab to select current, authorised policy evidence. Preserve source IDs and measure whether the required rule and exception are both present.
Use the agent tool lab for ownership checks, state validation and durable operation identity. Keep side effects in the local database while integrating the model.
The scripts are separate teaching components. Connecting them is part of the capstone, not an implied feature of the download. Keep each component's input and output contract explicit during integration.
Specify an integration sequence
A bounded implementation can follow this order:
- Classify intent and identify missing order information.
- Resolve the order through the authorised lookup.
- Retrieve the policy evidence required for that order state.
- Produce and validate an answer or proposed action.
- Execute an authorised action with a durable operation ID.
- Verify the resulting state and draft the final message.
If an earlier step fails, return its defined outcome. Do not ask a later drafting step to conceal the failure.
For a simple policy, calculate eligibility in code from the authoritative status. The model can explain the decision without becoming the authority for the rule.
Build an evaluation matrix
Include at least the following cases and review their expected results:
| Case | Expected behaviour |
|---|---|
| Eligible order, informational question | Explain eligibility without changing state |
| Eligible order, explicit action request | Cancel once and report the verified result |
| Dispatched order | Explain the applicable next process |
| Missing order ID | Ask a focused question |
| Another customer's order | Deny access without exposing record details |
| Missing policy exception | Identify insufficient evidence for the next step |
| Outdated policy in search candidates | Exclude it from active evidence |
| Instruction embedded in a source | Treat it as source content |
| Lost response after cancellation | Reconcile using the same operation identity |
| Model output with an invented citation | Reject or regenerate under a bounded policy |
Add realistic variations and separate development cases from the final review set. The table establishes coverage requirements; it does not determine the sample size needed for a production reliability claim.
Create an evaluation and cost report
Report task success, critical constraint violations, retrieval coverage, output contract failures and unnecessary abstention. Include per segment counts and repeated trial results where relevant.
Record the prompt, model, schema, tool and dataset versions. State whether outputs came from an actual model or an authored fixture.
Measure token usage and elapsed time on live runs. If offline fixtures were used, report those measures as unavailable rather than inventing them. Estimate monthly cost only with explicitly stated traffic and pricing assumptions.
Rehearse two operational incidents
First, make the retrieval service unavailable. Confirm that a failed lookup is not interpreted as evidence that no policy exists.
Second, deploy an intentionally poor prompt into the local release configuration, run the regression comparison and restore the previous version. Record which evidence triggered the rollback decision.
The rollback restores future behaviour. If the poor version already changed state, explain how those effects would be identified and reconciled separately.
Submit a release packet
The packet should contain the task contract, runnable prototype, version manifest, evaluation data, results, example traces and rollback instructions. Add a short decision explaining whether to release, revise or gather more evidence.
A reviewer should be able to answer these questions from the submitted evidence.
- Does the application solve the intended task?
- Are its actions bounded by the user's authority?
- Can its failures be diagnosed?
- Does the evidence justify the proposed exposure?
The course's multiple choice assessment measures understanding of these topics. This capstone is a separate practical exercise and is not automatically graded by that assessment. Its value is the implementation and evidence a practitioner can show.