LLM CONTROL PLANE READINESS CHECKLIST (v1)

Goal: Make one LLM feature observable, steerable, testable, and safe to change.

1) TRACEABILITY (Must-have)
- [ ] Every LLM call emits a trace ID linked to a user/session/request.
- [ ] Trace captures: provider, model name, major generation params, and latency.
- [ ] Trace captures prompt template ID/version (git SHA or registry version).
- [ ] Trace captures tool/function calls (name, inputs, outputs, errors).
- [ ] Logs are searchable by user ID, trace ID, and time window.

2) PROVENANCE FOR RAG (Must-have if you retrieve anything)
- [ ] Each retrieved chunk has immutable identifiers (source + doc ID + version).
- [ ] Trace stores the full retrieval set (IDs + scores + query text).
- [ ] You can reproduce retrieval for a past request (same index snapshot or recorded chunks).
- [ ] Index rebuilds are treated as releases (named version + changelog).

3) EVALS AS A RELEASE GATE
- [ ] You have a fixture set of real product examples (inputs + expected behavior).
- [ ] At least one deterministic check exists (schema validation, exact match, allowlist).
- [ ] You run evals on every change to prompts, tool schemas, retrieval settings, and routing.
- [ ] You store eval results by version so you can diff regressions.

4) ROUTING + FALLBACKS
- [ ] Requests can be routed by task (classification vs. drafting vs. extraction).
- [ ] You have a fallback model/provider plan for provider incidents.
- [ ] You have a “degraded mode” (template response, disable tool calling, or read-only).
- [ ] Rollback is a config change, not a redeploy.

5) POLICY + GOVERNANCE IN THE CALL PATH
- [ ] PII/secret handling is defined (redaction, blocking, or secure tool call).
- [ ] Output policy enforcement is explicit (what is blocked, what is rewritten, what is logged).
- [ ] Data retention rules are implemented (what’s stored, for how long, and why).
- [ ] You can produce an audit bundle for a single answer: trace + prompt version + sources + policy decisions.

6) PRODUCT UX FOR CONTROL (Not optional for serious users)
- [ ] Users can report a bad answer from the UI with context attached.
- [ ] Internal tooling lets support/eng pull the exact trace quickly.
- [ ] Admins can control sources (include/exclude) and see changes safely.

How to use this checklist:
- Pick ONE feature and get it to “mostly checked.” Don’t boil the ocean.
- The first milestone is reproducibility: the ability to explain one output end-to-end.
- Only after that, expand to more features and deeper automation.