Production Agent Readiness Checklist (Ops-First)

Use this checklist before you call anything “GA.” It assumes your agent can call tools (APIs) and may mutate state.

1) Scope and blast radius
- Define the single workflow the agent owns (start/end conditions). Write it in one paragraph.
- List every system it can touch (Jira, Salesforce, Stripe, Slack, email, data warehouse, etc.).
- For each system, mark actions as READ or WRITE. Default to READ.
- Define “stop conditions”: what errors or uncertainty should force escalation to a human.

2) Tool contracts (non-negotiable)
- Every tool has: strict input schema, strict output schema, and documented error codes.
- Tools that create/charge/send have idempotency keys and safe retry behavior.
- Tools support a dry-run mode where possible (return diff / proposed action).
- Rate limits and timeouts are explicit; failures are returned in a way the agent can classify.

3) Permissions and environments
- Separate credentials for dev/stage/prod.
- Tool allowlists per role (support agent vs finance vs engineering) and per tenant.
- Deny-by-default for sensitive data fields (PII) unless explicitly required.

4) Observability
- Trace each request end-to-end: model messages, retrieval results (if any), tool calls, tool outputs, and final action.
- Correlate traces with a request ID visible to operators.
- Alerts for: tool error spikes, unusual tool-call loops, and sudden cost/latency increases.

5) Evaluation gates
- Unit tests for tools (auth fail, validation fail, timeout, partial data).
- Scenario eval set for agent behavior with expected outcomes (not just “good tone”).
- Run evals on every prompt/tool/schema change in CI.
- Define regression rules: what failures block deploy.

6) Human-in-the-loop (where it matters)
- Human approval required for irreversible actions (payments, deletions, outbound customer emails, permission changes).
- Approval UI shows: proposed action, evidence (retrieved sources), and exact tool call arguments.
- Clear escalation routing: who gets paged or queued when the agent stops.

7) Rollback and incident response
- For each write tool, define the compensating action (undo, refund, revert config).
- Circuit breaker: a single switch to disable write tools globally.
- Runbook: how to investigate a bad action using traces; how to notify affected users if needed.

If you can’t check these boxes, you don’t have a production agent. You have a demo—and demos are allowed, as long as everyone calls them what they are.