AI SUPPLY CHAIN READINESS CHECKLIST (STARTUP EDITION)

Use this to pressure-test whether you’re shipping a demo or an operable product.

1) DATA BOUNDARIES (WRITE IT DOWN)
- List every data type you ingest (PII, PHI, financial, source code, contracts, support tickets).
- For each type: where it’s stored, who can access it, and how deletion works.
- Document your stance on model training/retention with every upstream provider (what is logged, for how long, and how to opt out if available).
- Define a redaction rule for logs (what never gets written to disk).

2) RETRIEVAL + PERMISSIONS
- If you do RAG: confirm index permissions match application permissions.
- Store retrieval metadata: document IDs, timestamps, source system, and access policy.
- Require citations for any answer that claims a fact from customer data.

3) EVALS THAT BLOCK REGRESSIONS
- Pick one revenue-critical workflow and create a “golden set” of inputs/expected outputs.
- Define pass/fail metrics that map to the job (exact fields extracted, required steps completed, policy violations = zero).
- Add an automated eval run to CI on every prompt/model/retriever change.
- Establish an incident workflow: how you triage a bad output, and how it becomes a new test.

4) OBSERVABILITY
- Capture traces per request: model/provider, prompt version, retrieved context identifiers, latency, and failure mode.
- Track cost drivers qualitatively if exact accounting is hard: which routes are expensive, which calls retry often.
- Maintain an error taxonomy (timeouts, tool failures, hallucination-like behavior, retrieval miss, parsing failure).

5) RUNTIME CONTROLS
- Implement timeouts and retries with backoff.
- Add at least one fallback path: smaller model, cached response, or a non-AI deterministic mode.
- Add caching where safe (especially for repeated retrieval or deterministic transformations).

6) PROCUREMENT-READY ARTIFACTS
- Maintain a vendor list (model APIs, vector DB, observability, hosting) and what data each sees.
- Draft a security one-pager: data flow diagram, logging policy, retention, and access controls.
- Maintain an “answer pack” for common security review questions (SOC 2 status if applicable, encryption approach, subprocessors).

7) MODEL CHURN PLAN
- Keep prompts and schemas versioned.
- Maintain a model swap playbook: run eval suite, compare outputs, review deltas, then ship.
- Decide in advance what you do during provider outages (rate limits, partial degradation, queueing, or fail closed).

If you can’t complete this checklist for one workflow in a week, scaling growth will scale chaos. Start with one path, make it provable, then expand.