Agentic AI Production Readiness Checklist (2026 Edition) Use this before giving an AI agent access to real tools (CRM, billing, email, ticketing, code repos). Goal: measurable task success, bounded risk, predictable operating cost. 1) Workflow Definition (quick but strict) - Write a one-sentence workflow contract (input → outcome). Example: “Given a Zendesk ticket, draft a policy-correct reply with citations to billing records and policy docs.” - Classify action types the agent may attempt: read-only, write, irreversible (refunds, deletions), external communications (email/SMS). - Define success signals you will track: task success rate, policy violation rate, escalation/hand-off rate, latency, cost per run. 2) Context & Tooling (platform work) - Standardize tool interfaces (MCP-style connectors or one internal tool schema). - Require typed parameters and structured outputs (JSON) and document failure modes per tool. - Enforce least privilege per tool: separate read vs write scopes; staging vs production credentials. - Add idempotency keys for write operations to prevent duplicate side effects. 3) Evaluation Gates (release discipline) - Build an initial eval set from real cases. Include edge cases: VIP customers, refunds, angry users, missing data, tool outages. - Maintain three eval tiers: a) Task success (deterministic checks where possible) b) Safety/policy (PII leakage, prompt injection, forbidden actions) c) Operational behavior (max tool calls, max latency, loop and retry-storm detection) - Set ship/rollback rules that block releases when regressions appear. 4) Safety & Governance (non-negotiable) - Log every tool call with enough detail to replay: user identity, timestamp, tool name, parameters, outputs, model version, prompt hash. - Put approvals in front of high-impact actions (refunds, contract edits, outbound sends, data deletion). - Handle PII and secrets deliberately: redact before model calls; keep secrets out of prompts; use a vault. - Require provenance for user-facing outputs (citations to docs and tool outputs; verification step that forbids new facts). 5) Cost & Performance (protect margins) - Meter tokens and cost per step and per customer/account. - Route work: small model for classification/extraction; frontier model for planning and final decisions. - Add caching: prompt/result caching, semantic caching, retrieval caching for stable corpora. - Set budgets: max tokens per run, max tool calls, max retries per step; alert on breaches. 6) Operations & Incident Response - Build dashboards that match reality: success rate, violation rate, latency (including tail latency), cost per run, tool error rate. - Add circuit breakers: auto-disable write actions if violations rise or critical tools degrade. - Define human escalation paths: when uncertain, the agent hands off with context and a proposed next step. - Run game days regularly: simulate tool outages, permission denials, and prompt injection attempts. Exit Criteria (go/no-go) - You can replay any agent run end-to-end from logs quickly. - You can show least-privilege access for every tool the agent can touch. - Evals run in CI/CD and block releases when thresholds fail. - Write actions are bounded by approvals, budgets, and circuit breakers. - Unit economics are understood: cost per successful outcome makes sense for your pricing.