LLM Routing + Distillation Release Checklist (Operator Edition)

Use this to stop reflexive fine-tuning and start shipping predictable quality.

1) Define the request classes (you can’t route what you can’t name)
- List 5–10 top workflows by traffic or revenue impact.
- For each, mark: risk (PII/finance/medical/legal), tool need (DB/search/code), tolerance for latency, and required output format.
- Write a “failure is unacceptable” line for each workflow (e.g., wrong refund policy, wrong account balance, unsafe instructions).

2) Build a minimal eval set before you change anything
- Gold set: 20–50 real examples per workflow (sanitized).
- Adversarial set: prompt-injection attempts, malformed inputs, conflicting instructions, empty context.
- Checks: schema validity, citation requirement (if RAG), refusal behavior, and correctness where you can verify against a source of truth.

3) Pick your first router policy (start blunt)
- Default to a cheaper/faster model for low-risk, low-ambiguity requests.
- Escalate to a stronger model when:
 - The user’s request is high-risk.
 - Required schema fails validation.
 - Retrieval returns low-quality context (missing key docs).
 - The model expresses uncertainty (you can prompt for a confidence label) OR the request is underspecified.
- Log every escalation with the trigger reason.

4) Decide between test-time compute and distillation
- Use test-time compute when quality is needed on the long tail:
 - Add verifier pass, multi-sample + rank, or tool calling.
- Use distillation when the task is stable and high-volume:
 - Define a strict output contract.
 - Generate teacher outputs with a stronger model.
 - Train a smaller model to imitate the teacher only for that contract.
- Keep escalation even after distillation (out-of-distribution is guaranteed).

5) Fine-tune only if you can prove the bottleneck is behavior, not system design
- You have passing retrieval.
- You have schema gates.
- You have router escalation.
- Your eval failures persist and are consistent across prompts and context.

6) CI gates (block releases on regressions)
- Contract gate: schema validity must not regress.
- Safety gate: refusal/PII handling must not regress.
- Retrieval gate: citations required; answers must reference retrieved sources (if RAG).
- Router gate: misroutes on gold set must not increase.
- Cost/latency gate: token/tool-call budgets must stay inside your limits.

7) Operational hygiene
- Version everything: prompts, retrieval config, router rules, model IDs.
- Add rollback paths: pin model versions where possible; keep previous prompt/router config ready.
- Monitor live traffic for: schema failures, escalation rate changes, and spikes in tool-call errors.

If you can’t enforce these gates, don’t fine-tune. Fix your system first.