Agent Control Plane Shipping Checklist (v1)

Use this to ship an agent that can take actions (write operations) without turning your product into an incident generator.

1) Define the workflow boundary (be brutally specific)
- Name the workflow (e.g., “Create Jira ticket from support conversation”).
- List allowed actions. List explicitly forbidden actions.
- Identify the systems touched (Jira, Slack, email, GitHub, CRM).
- Decide what “done” means in product terms (fields populated, links attached, status set).

2) Identity + permissions (treat the agent as a user)
- Create an agent identity that is not your backend’s admin credential.
- Ensure actions are attributable: “Agent A acting on behalf of User U in Tenant T.”
- Implement least-privilege tool scopes (allowlist tools + operations).
- Add a per-tenant connector permission screen (admins must be able to revoke access).

3) Approval and staging (plan → review → execute)
- Require the agent to output a structured plan before any write.
- Decide approval rules:
 - Always approve for irreversible actions (refunds, deletes, exports).
 - Optional approval for low-risk writes (drafts, comments) based on customer settings.
- Build a diff view for writes (what will change, where, and why).
- Add an emergency kill switch (per workflow and per connector).

4) Observability (make every run explainable)
For every agent run, record:
- Run ID, timestamp, tenant, user, agent identity.
- System prompt/instructions version.
- Model identifier and key generation settings.
- Retrieval inputs/outputs: document IDs, URLs, hashes, or snapshots.
- Tool calls: name, parameters, responses, errors.
- Final output shown to user.

5) Replay and incident readiness
- Ensure you can reconstruct a run from stored artifacts.
- Implement rate limits and circuit breakers for repeated tool failures.
- Build a “support view” that a human can use without engineering help.

6) Evaluation as regression (tie to releases)
- Maintain a versioned test set per workflow (gold cases).
- Run evals on:
 - Prompt/instruction changes
 - Retrieval/index changes
 - Tool schema changes
 - Model/provider changes
- Gate rollout with a canary or tenant opt-in.

7) Undo story (don’t ship without one)
- For each write action, define how to reverse it.
- Prefer idempotent operations and compensating transactions.
- If undo is impossible (third-party limitation), require explicit approval and show a warning.

8) Customer-facing controls (sell trust)
- Admin settings: tool access, approval thresholds, data sources.
- Audit log export.
- Explainability: “why” with citations to records used.
- Clear UI states: Draft vs Executed vs Failed vs Rolled back.

If you can’t complete sections 2–7 for a workflow, keep the agent in read-only mode. That’s not a compromise; it’s competence.