Agentic AI Production Readiness Pack (2026 Edition)

Use this pack to move from “cool demo” to “approved for production.” It’s built for a small startup team shipping an agent that can take real actions (create/update records, send messages, trigger workflows).

1) Define a measurable, narrow job
- Write a one-sentence job statement: “The agent completes X type of request end-to-end.”
- List explicit exclusions (what it must never do).
- Pick 1 primary KPI and 2 guardrail KPIs.
 Example primary KPI: share of eligible requests resolved without human intervention.
 Guardrails: unsafe-action rate; average cost per run.
- Set a target and a baseline using recent production data.

2) Permissions and tool access (least privilege)
- Inventory every tool/API the agent can call.
- Create scoped roles (read vs write; per-project/per-queue constraints).
- Add allowlists for destinations (domains, projects, record types).
- Implement revocation and a global kill switch.
- Define approval thresholds for higher-risk actions.

3) Build your evaluation suite before scaling
- Collect a representative task set (happy path + edge cases + hostile inputs).
- Label outcomes: success, safe-fail, unsafe-fail.
- Track: task success rate, tool-call correctness, escalation rate, and timeouts.
- Add release gates (block deploy when core tasks regress).
- Schedule a weekly eval review with engineering + product.

4) Observability and audit trail (non-negotiable)
- Log: model name/version, prompts, retrieved context references/IDs, tool inputs/outputs, and decisions.
- Redact PII in logs; store references to source docs instead of copying content.
- Capture tokens and cost per run.
- Provide an exportable trace customers can hand to security/compliance.

5) Cost and latency budgets
- Define max runtime, max retries, max tool calls, and max spend per run.
- Add routing: smaller model for classification; stronger model for final output.
- Cache repeated answers and deterministic lookups where safe.
- Create alerts for budget breaches and spend anomalies.

6) Pilot rollout template (30 days)
Week 1: Connectors + permissions + baseline metrics + shadow mode.
Week 2: Limited write actions with approvals; start measuring outcomes.
Week 3: Expand eligibility; tighten policies based on failure analysis.
Week 4: Present KPI results, incident log, and a scaling plan.

7) Contract and pricing guardrails
- Prefer: platform fee + metered outcomes (with volume tiers).
- Put data handling in writing: retention period, training usage (yes/no), deletion process.
- Include incident response commitments: notification window and escalation contacts.

Exit criteria for production
- Core task success meets target across consecutive evaluation windows.
- Unsafe-fail rate stays below an explicit, agreed threshold.
- Trace logging coverage stays consistently high.
- Customer security signs off on permissions, retention, and kill switch.

If you can’t meet the exit criteria, narrow the job. Agent success in 2026 comes from governed, measurable execution—not maximum autonomy.