All articles
- Engineering
- AI agents
What production-grade AI actually means
Demos are easy. Here is the checklist we use before an AI system is allowed to touch real work.
Hassan Mujtaba · Oct 6, 2026 · 1 min read
A prototype that works on ten examples is not a product. Before anything we build goes live, it has to pass this checklist.
Reliability
- Retries and timeouts around every model and API call
- Structured outputs validated against a schema
- A fallback path when the model is unsure
Guardrails
- A human approval step for high-impact actions
- Allow-lists for the tools an agent may call
- Evaluation sets that run on every change
Security
- Role-based access, and secrets kept out of code and prompts
- Your data is never used to train third-party models
- Audit logs of what the system did and why
Cost control
- The smallest model that passes the evals
- Caching of repeated work
- Usage dashboards and budget alerts
Operability
- Monitoring and alerting
- Documentation and a hand-over your team can own
If a vendor can't show you how each of these is handled, you're buying a demo.
