LLM PRODUCTION READINESS CHECKLIST (OPS-FIRST)

Use this to harden one workflow (not your whole product at once). The goal is boring reliability: predictable outputs, bounded actions, and fast rollback.

1) Define the workflow boundary
- Name the workflow (e.g., “refund eligibility triage”).
- List the allowed inputs (user message, ticket metadata, account flags).
- List the allowed outputs (one of N decisions + explanation + required fields).
- Decide what the model is NOT allowed to do (e.g., “never issue a refund,” “never modify CRM records”).

2) Output contract (no free-form parsing)
- Choose a structured format (JSON) and define a schema.
- Add deterministic validators:
 - Schema validation (required fields, enums).
 - Business rules (e.g., refund amount must be <= purchase amount).
 - Safety rules (no secrets, no credentials, no policy-banned content).
- Define failure behavior: reject + retry, or route to human.

3) Tool governance (treat tools like production APIs)
- Inventory every tool call the model can make.
- Split tools into READ vs WRITE.
- Enforce least privilege (scoped tokens, service accounts).
- Add an allowlist (which tools are callable in this workflow).
- Add friction for side effects:
 - confirmations (“Are you sure?”),
 - dual-control approvals,
 - rate limits,
 - idempotency keys.

4) Versioning and release discipline
- Put prompts, tool schemas, and model config under version control.
- Require code review for changes.
- Tag releases so you can map any production run to an exact version.
- Predefine rollback steps (including how to revert tool permissions if needed).

5) Evals as a release gate
- Create a small, curated eval set for the workflow.
- Include:
 - standard cases,
 - edge cases,
 - at least one adversarial/prompt-injection attempt.
- Define pass/fail criteria (schema failures and policy violations should fail hard).
- Run evals in CI on every prompt/model/tool change.

6) Tracing and audit logs
- For each run, record:
 - prompt version, model, temperature/config,
 - input payload (with redaction rules),
 - tool calls and tool outputs,
 - validator results,
 - final decision and latency.
- Ensure logs are access-controlled and retention is defined.

7) Incident playbook (write it before you need it)
- What triggers an incident? (e.g., policy violation spike, tool-call anomaly)
- Immediate actions:
 - disable WRITE tools,
 - rollback to last known-good version,
 - route to human fallback.
- Who is on-call, and who can approve re-enabling actions?

Pick one workflow this week and complete sections 1–3. If you can’t bound outputs and tools, don’t ship the feature as autonomous.