Agent UI Shipping Checklist (Runs, Approvals, Audit Trails)

Use this to review any AI feature that can read sensitive data or take actions in external systems.

1) Define the unit of work
- Name the “run”: a bounded attempt to complete a task (e.g., “Create Jira issue drafts from this Slack thread”).
- Define the artifacts the run must produce (draft ticket, PR, document, record update). If there’s no artifact, it’s probably a demo.

2) Choose an execution mode
- Suggest-only: agent proposes actions, never writes.
- Drafts: agent writes drafts that require approval.
- Auto-execute: agent writes directly. Only acceptable for low-risk, reversible actions.
- For each tool, write down what counts as an “external side effect” (email send, publish, delete, payment, permission change).

3) Identity and permissions
- Decide whether the agent acts as: (a) the user via delegation, (b) a service account, or (c) a shared team agent.
- Ensure the UI shows the actor identity for every run.
- Apply least privilege per tool integration (scopes/roles). Avoid broad “admin” scopes by default.

4) Context control and provenance
- Provide a visible list of sources used (documents, tickets, messages, links).
- Let users add/remove sources before execution.
- Store source references in the run log so results are explainable later.

5) Approval gates and previews
- Add an approval step before any irreversible or external action.
- Show a preview that matches the target system: diffs for text/code, field-level changes for records, and a final “what will happen” list.

6) Reversibility
- For every write action, define an undo path: revert diff, delete created object, rollback field changes, or create compensating actions.
- Put the undo link in the run history.

7) Run history, audit, and replay
- Persist a run log with: inputs, selected context, tool calls, approvals, outputs, timestamps.
- Make it searchable.
- Add “replay with changes” so users can rerun with updated inputs without starting from scratch.

8) Failure handling and escalation
- Decide what happens on: missing permissions, tool downtime, ambiguous user request, low-confidence extraction.
- Provide a one-click handoff to a human workflow (create a Jira ticket, open a Zendesk case, post to Slack channel) with the run trace attached.

9) Safe defaults
- Rate limit outbound actions (email, invites, bulk updates).
- Use allowlists for domains, recipients, repos, or projects where appropriate.
- Start new integrations in sandbox/test mode when possible.

10) Launch criteria (don’t skip)
- A skeptical operator can answer: what data was used, what actions were taken, what changed, how to undo, and how to prevent repeats.
- If you can’t meet that bar, don’t ship “chat.” Ship the run log and approvals first.