Agentic Workflow Launch Checklist (2026)

Use this to ship a tool-using agent that completes real work (tickets, refunds, PRs, account changes) without turning your company into an incident factory.

1) Pick the right first workflow
- The output must be an artifact with an ID (ticket ID, PR number, refund ID, invoice ID).
- Success must be binary or close to it (created/updated/closed, correct/incorrect fields).
- Avoid “answer questions about docs” as the first workload; it’s hard to score and easy to argue about.

2) Define the system of record and state
- Write down which system owns truth (Jira, GitHub, Stripe, Salesforce, Postgres).
- Decide where workflow state lives (a workflow engine like Temporal, or an existing orchestrator you already trust).
- Ensure every step is idempotent or has a compensation action (undo/rollback plan).

3) Design tools with strict schemas
- List 5–10 allowed tools (API calls) the agent can use.
- For each tool: name, description, JSON schema for arguments, and explicit failure modes.
- Implement deny-by-default routing: unknown tool requests fail closed.

4) Permissions and approvals
- Use least-privilege credentials for tool execution (scoped service accounts, limited roles).
- Add explicit approval gates for: payments/refunds above your threshold, customer-facing messages, production changes, data deletion, and access grants.
- Never give the agent standing admin privileges “for convenience.”

5) Evals before launch
- Create a versioned JSONL eval set: at least 30 cases across happy path, missing context, ambiguity, and hostile requests (e.g., “delete prod”).
- Score what matters: correct tool choice, valid arguments, policy compliance, and abstention when context is missing.
- Run evals in CI on every prompt/tool/schema/model change; store results with version identifiers.

6) Observability you can use
- Log every tool call with request ID, tool name, validated args, response status, and resulting artifact IDs.
- Store the prompt/tool schema versions tied to each run.
- Build a simple dashboard: error rate, deny events, approval queue size, and top failing eval cases.

7) Rollout plan
- Start in shadow mode: agent proposes actions; humans execute; compare outcomes.
- Canary: enable for a small user group or low-risk segment.
- Add a kill switch and a rollback strategy (model pinning + config revert).

8) Operating discipline
- Assign an owner for eval maintenance and incident response.
- Treat model upgrades like dependency upgrades: staged rollout, eval gate, and rollback.
- Review deny logs and failure traces weekly; turn them into new eval cases.