AgentOps Readiness Checklist (2026 Edition) Use this checklist before you increase autonomy (allowing write actions, customer-facing decisions, or unattended execution). Score each item as: Not Started / Partial / Done. If any Safety & Governance item is Not Started, do not expand autonomy. 1) Workflow & ROI Definition - Pick one primary workflow (not a general assistant). Document start state, end state, and abort conditions. - Choose a small set of business KPIs (examples: deflection, time-to-resolution, CSAT movement). - Write down real high-severity error examples (privacy exposure, incorrect refund, incorrect account change, compliance breach). - Define an error budget by severity (high, medium, low) and what happens when you exceed it. 2) Data & Grounding - Identify authoritative sources and name the “source of truth” owner for each. - Require citations (or evidence links) for factual, policy, pricing, or compliance-related claims. - Add freshness rules for time-sensitive topics (pricing, security policies, contracts) and define what “stale” means for your business. - Document excluded data classes (HR data, sensitive customer data, regulated categories, specific PII types). 3) Tooling & Permissions (Least Privilege) - Separate read-only tools from write tools. - Enforce structured tool inputs (JSON schema) with server-side validation. - Use idempotency keys for write actions to prevent duplicate execution on retries. - Use scoped, short-lived credentials; map access to IAM roles and groups. - Require approval tokens for high-impact actions (bulk messaging, large refunds, deletions, permission changes). 4) Evaluation & Testing - Build a labeled “golden set” from real requests and real edge cases. - Add adversarial tests: prompt injection, jailbreak attempts, ambiguous instructions, tool misuse patterns. - Track weekly: task success, citation quality, policy adherence, escalation quality, severity-weighted failures. - Gate releases: no ship if quality regresses beyond your defined threshold. 5) Observability & Incident Response - Implement tracing across: prompt → retrieval → model calls → tool calls → response. - Log: model/version, prompt template version, retrieved doc identifiers, tool parameters (redacted), latency, and cost estimates. - Provide replay tooling for failed runs (same inputs, and the same tool outputs when feasible). - Write incident runbooks: rollback steps, disable write tools, switch to safe mode, notify security, preserve evidence. 6) Cost & Latency Control - Define budgets per workflow (cost per completed task and latency targets for interactive vs background runs). - Implement routing: smaller models for extraction/classification; stronger models for complex reasoning. - Cache where it’s safe (retrieval results, embeddings, deterministic transforms). - Monitor anomalies: token spikes, tool-call loops, repeated retries, and unusual request patterns. 7) Human-in-the-Loop Operations - Define escalation triggers (missing citations, low confidence, policy conflicts, tool failures, sensitive topics). - Give operators a clear UI: traces, sources, tool calls, and the agent’s stated rationale. - Capture feedback with reason codes and a way to provide the corrected outcome. - Schedule audits and track audit pass rates by workflow and severity. Exit Criteria (Ready to Expand Autonomy) - Eval coverage includes the most common intents and the highest-risk scenarios, with stable results across multiple runs. - Observability supports end-to-end replay and root-cause analysis quickly enough for on-call response. - Write actions are protected by policy checks and approvals, and audit logs are complete. - Cost and latency dashboards exist per workflow; routing is live; budgets are met consistently. Rule for expansion - Expand autonomy in one dimension at a time (scope, permissions, or volume)—never all three at once.