ICMD AI Agent Production Readiness Checklist (2026 Edition)

Use this checklist to move an AI agent from pilot to production without getting stuck in demo-ware.

1) Pick the first workflow (scope)
- Choose a single unit of work (one ticket, one invoice, one alert) with a clear DONE definition.
- Prefer high-frequency work so learning compounds over time.
- Bound the downside: avoid irreversible actions at launch (payments, deletions, account closures).
- Identify a real owner with budget and KPIs (Controller, Head of Support, SecOps lead).

2) Define success metrics (outcomes + reliability)
- Outcome KPI: backlog reduction, cycle time, time-to-close, or a domain KPI the operator already tracks.
- Reliability KPI: task success rate (TSR), tool-call error rate, silent failure rate.
- Human effort KPI: interventions per 100 tasks; typical review time.
- Cost KPI: cost-to-serve per unit; inference and tooling as a share of revenue.

3) Build guardrails before autonomy
- Least-privilege permissions (scoped OAuth, RBAC).
- Hard limits: max tool calls, max cost per task, timeouts.
- Approval workflow for any write action; step-up verification for sensitive steps.
- Escalation rules: low-confidence threshold and clear routing to humans.

4) Instrument everything (audit + observability)
- Structured logs per task: inputs, tool calls, outputs, policy decisions.
- Tracing for latency and cost; dashboards for a failure taxonomy.
- Exportable audit reports for compliance and security reviews.

5) Evaluation and rollout plan
- Build an eval set from real history and keep it versioned.
- Gate changes: no prompt/policy/model updates unless evals pass.
- Roll out in phases: shadow mode → assist mode (human approval) → partial autonomy.
- Define rollback: disable tools or revert versions quickly.

6) Pricing and packaging (make it legible)
- Price around outcomes (per resolved unit) plus a platform fee.
- Publish what counts as billable and what doesn’t (retries, failed tasks, rework).
- Design for margins: route models intelligently and cap runaway costs.

7) Security & compliance readiness
- Data retention policy and customer controls.
- Clear separation of customer data across tenants.
- Incident response process: escalation contacts, postmortems, and response expectations.

Exit criteria for “production”
- High TSR on low-risk workflows or predictable safe escalation.
- Near-zero silent failures.
- Complete action logging with export.
- Clear unit economics and a named operator owner.