AI SYSTEM RELEASE GATE CHECKLIST (2026)

Use this as a release gate for any AI workflow that touches revenue, customer data, or production operations.

1) Define the unit you are shipping
- Name the workflow (e.g., “refund assistant,” “sales email drafter,” “on-call runbook agent”).
- List the tools it can call (Stripe, Salesforce, internal APIs) and what permissions it needs.
- Write a one-sentence success condition that an operator can judge.

2) Create an eval dataset before you tune anything
- Collect real scenarios: support tickets, transcripts, common edge cases, past failures.
- For each scenario, store: input, expected properties (not just “exact answer”), and any required sources.
- Tag scenarios by risk (low/medium/high). High-risk scenarios require stricter gating.

3) Version the behavior surface
- Prompt template + system message version.
- Model identifier (provider + model name/version) and decoding settings.
- Retrieval config: embedding model, chunking settings, top-k, filters.
- Index snapshot/version: document set timestamp and deletion semantics.
- Tool schemas and tool error handling behavior.

4) Run end-to-end evals (not single-call demos)
Minimum checks:
- Regression: new build vs last-known-good on the same dataset.
- Tool failures: simulate timeouts, 4xx/5xx responses, partial outages.
- Retrieval: confirm required sources are actually retrieved for key questions.
- Policy: run a small adversarial set (PII leakage, forbidden actions, prompt injection attempts).

5) Add trace requirements (block release if missing)
Each production trace must include:
- request_id, user/session context (as allowed), model id, prompt version
- retrieved doc ids (and scores if available)
- tool calls with status, latency, and errors
- final output (or a reference to stored output)

6) Canary + rollback plan
- Define canary population (internal users or traffic slice).
- Define monitoring: tool error rate, latency, blocked outputs, fallback rate.
- Pre-stage rollback: last-known-good prompt/retrieval/tool config that can be restored without a code redeploy.

7) Ownership
- Name the DRI for: prompt changes, retrieval/index changes, tool schema changes, and policy changes.
- Add an incident playbook: what to disable first (tools? retrieval? entire agent?) and how to communicate impact.

If you can’t pass this checklist for a workflow, it’s not ready to be an “agent.” It’s a prototype.