COMPOUND AI SHIPPING CHECKLIST (2026 EDITION) Use this before launch and as an ongoing operations list for production AI systems (RAG, agents, copilots). 1) WORKFLOW + RISK DEFINITION - Name the workflow (examples: “support resolution”, “refund handling”, “PR generation”). - Define success using a business metric you already trust (containment, handling time, acceptance rate, time-to-resolution). - Assign a risk tier: LOW (drafting), MED (user-facing advice), HIGH (money movement, auth, legal commitments). - For HIGH risk: require structured outputs and an explicit verifier step (model or deterministic checks) before any action runs. 2) ROUTING + BUDGETS (COST + LATENCY) - Implement an intent router (rules or a small model) with explicit fallback rules. - Set budgets: max tool calls, max tokens, max wall-clock time, max retries. - Track cost per successful task, not just cost per request. - Cache stable outputs (policy summaries, repeat troubleshooting steps) with a TTL and a clear invalidation rule. 3) PRIVATE DATA PLANE (GOVERNED CONTEXT) - Inventory data sources (docs, CRM, tickets, code, BI) and set a freshness expectation per source. - Enforce permissioned retrieval: filter by identity and document/row ACLs before any context is sent to a model. - Log retrieval for every request: request_id, document IDs, chunk IDs, timestamp, user identity. - Set retention and deletion policies for prompts, outputs, and embeddings. 4) TOOL SAFETY (AGENTS) - Create a tool allowlist per intent. Deny-by-default. - Require structured outputs for actions (schema-validated) and validate before execution. - Add a verifier step for high-risk actions. - Implement safe failure states: explicit “cannot complete” plus an escalation path. 5) EVALS + RELEASE ENGINEERING - Build a golden set that covers normal requests and edge cases. - Add adversarial cases: prompt injection, data exfiltration attempts, permission bypass attempts. - Define pass/fail gates: groundedness, citation quality, refusal correctness, tool safety. - Use shadow traffic before ramping; define rollback triggers tied to user impact and safety violations. 6) OBSERVABILITY + ON-CALL - Instrument traces across retrieval, model calls, tool calls, and final output. - Monitor tail latency, tool error rate, misroute rate, refusal rate, and cost per successful outcome. - Maintain an on-call playbook: how to disable a tool, adjust routing, or roll back prompts/models. 7) SECURITY + COMPLIANCE READINESS - Document the data flow: where data moves, where it’s stored, and who can access logs. - Provide customer controls where applicable: retention settings, audit export, residency options. - Schedule regular reviews with Security and Legal for high-risk workflows. If you can’t answer “What did the system retrieve, which tools did it call, and which rule allowed it?” it’s not ready for production.