Agent Runtime Release Gate Checklist (v1)

Use this as a release gate for any AI agent feature that can read customer data or take actions in external systems. If you can’t check an item, you’re shipping risk.

1) Identity, Permissions, and Secrets
- Define the agent’s identity model: does it act as the user, a service account, or both?
- Remove broad, long-lived API keys from prompts and model-visible context.
- Issue scoped credentials per tool call (least privilege). Scopes must match the action (read vs write).
- For irreversible actions (send external email, delete records, submit payments, change entitlements), require explicit user approval.
- Log every tool call with: user/org, scope, tool name, parameters, and outcome.

2) Tooling Contracts (Make Actions Boring)
- Each tool has a strict schema (inputs/outputs). Reject non-conforming outputs.
- Enforce idempotency for tools where retries are possible (create vs update semantics).
- Implement timeouts and bounded retries. No infinite “try again” loops.
- Add a “dry run” mode for high-risk workflows.

3) Tracing, Replay, and Auditability
- Every agent run produces a trace ID.
- Store a replayable snapshot: prompts, tool calls/results, model/provider identifiers, timestamps.
- Redact secrets and sensitive fields in logs; document what is redacted and why.
- Make it possible to answer: what data did the agent see, what actions did it take, and what safeguards were applied?

4) Retrieval (RAG) and Data Governance
- Retrieval must enforce source permissions (document-level and, if relevant, row-level).
- Store citations (document IDs / chunk IDs) for any answer that references retrieved content.
- Define retention and deletion: where embeddings live, how deletions propagate, and how caches are purged.
- Prevent cross-tenant retrieval by construction (separate indexes or mandatory tenant filters).

5) Evaluation and Regression Control
- Create an eval suite from real failures and adversarial inputs (prompt injection, malicious docs, weird formatting).
- Run evals on any change to prompts, tools, model versions, retrieval settings, or routing.
- Track policy violations separately from “quality” issues.
- Add canary releases for high-impact changes.

6) Cost, Rate Limits, and Degradation
- Enforce per-org budgets and per-workflow limits.
- Implement caching for repeat queries where safe.
- Define fallback behavior: cheaper model, reduced context, or human handoff.
- Alert on anomalies: spikes in tool calls, retries, token usage, or failure rates.

Shipping Rule:
If you can’t produce a trace that explains a bad outcome end-to-end—and show which permission or approval gate should have blocked it—don’t ship the automation. Ship a draft + approval step instead.