Agentic Ops Production Readiness Checklist (2026)

Use this checklist before an AI agent is allowed to execute real actions (emails, refunds, deploys, data updates). Mark each item: Not started / In progress / Done.

1) Scope & success criteria
- Pick one workflow with a measurable outcome (example: triage tier-1 tickets; create Jira issues with correct routing).
- Define success using quality + time + cost (Task Success Rate target; Human Intervention Rate target; Cost per Successful Task ceiling).
- Write a short “never do” list (example: delete customer data; email external recipients without approval).

2) Identity & permissions
- Give the agent a distinct identity (never a shared human token).
- Enforce least-privilege scopes per tool (separate read-only roles from write roles).
- Prefer short-lived credentials; rotate secrets; log auth failures and denied attempts.
- Implement action tiers (read-only, draft, low-risk write, high-risk write, production control).

3) Tool contracts & safety
- Define typed tool schemas (inputs/outputs) and validate outputs deterministically.
- Make side-effect tools idempotent (idempotency keys) and support dry-run.
- Add hard post-checks (amount limits, allowed recipients/domains, environment locks).
- Require approval gates for high-risk actions (thresholds, two-person rule).

4) Evals & testing
- Build an offline eval set from real work examples (enough variety to catch regressions).
- Include adversarial cases: ambiguous requests, missing data, policy violations, prompt injection.
- Run shadow mode on real inputs long enough to compare against human outcomes.
- Version prompts, tools, policies, and model settings so you can reproduce results.

5) Observability & replay
- Capture structured traces: model version, prompts, retrieved sources (or hashes), tool calls, tool responses, policy decisions, and final actions.
- Redact sensitive fields (PII, secrets) in logs; set retention by risk tier.
- Implement replay for incident investigation and debugging.
- Monitor: Task Success Rate, Human Intervention Rate, Policy Blocks, and Cost per Successful Task.

6) Cost controls
- Set budgets by workflow (spend caps and anomaly alerts).
- Add graceful degradation: route to smaller models, reduce retrieval depth, or require human approval under load.
- Attribute spend to outcomes (cost per resolution/merge/incident handled correctly).

7) Launch gates
- Start with Tier 0–1 autonomy (read-only/draft) before Tier 2+.
- Require explicit reliability and cost targets before expanding scope.
- Run an incident tabletop exercise (wrong email, wrong refund, infinite loop, data exposure).
- Document an agent kill switch and name on-call ownership.

If you cannot answer: “What did the agent do, who approved it, and what did it cost?” it is not ready for production autonomy.