Why demos lie
A carefully chosen demo prompt tells you almost nothing about production readiness. Real users send messy input, edge cases break assumptions, and models hallucinate under pressure.
We never ship an agent without an eval suite built from real (anonymized) client data.
What we measure
Accuracy on held-out test cases. Refusal rate on out-of-scope requests. Tool call correctness. Latency at p95. Cost per successful completion. Regression against previous model versions.
Each metric has a threshold. If the agent doesn't pass, it doesn't ship — regardless of how good the demo looked.
Continuous evals post-launch
Production is not the finish line. We sample live runs, flag low-confidence outputs, and re-run eval suites when models or prompts change.
This is how you catch drift before your operators do.