Retrieval-First AI Readiness Checklist (2026)

Use this before you debate “which model” or start fine-tuning.

1) Data sources & connectors
- List the exact systems you will index (e.g., Google Drive, SharePoint, Confluence, Slack, Jira, GitHub).
- For each source: define the sync method (webhook/event-driven if available; otherwise polling) and the freshness expectation (“how long can results be stale before users lose trust?”).
- Define deletion behavior: what happens when a doc is deleted, access is revoked, or a user leaves the company.

2) Permissioning model (no hand-waving)
- Write down the permission semantics you must match (groups, nested groups, link-sharing, external collaborators).
- Choose one approach and document it:
 a) Replicate ACLs into your index with auditable mappings, or
 b) Enforce authorization at query time via the source-of-truth.
- Create an explicit test plan: a small set of “can/can’t see” scenarios that must always pass.

3) Retrieval quality (relevance you can debug)
- Decide on vector-only vs hybrid (keyword + vector). If hybrid, define how you combine scores.
- Define chunking rules per document type (policies, runbooks, tickets, code, chat). Avoid one-size-fits-all.
- Require provenance: every retrieved chunk must carry a stable doc_id, version, and URL.

4) Grounding UX
- Make citations clickable and useful: title, snippet, and where it came from.
- Provide a “show sources only” mode as a fallback when generation is disabled or uncertain.
- Add a “why this result” debug panel for internal users (even if it’s hidden in production).

5) Safety & prompt injection posture
- Treat retrieved text as untrusted input. Add rules to ignore instructions found inside documents.
- Define a refusal policy: what the assistant will not answer, and how it explains that.
- Decide how you detect and handle sensitive data (PII/secrets) in both queries and retrieved context.

6) Evaluation & operations
- Build a seed set of real queries (from tickets, search logs, or interviews) and keep it versioned.
- Add regression checks for top intents whenever you change chunking, embeddings, ranking, or prompts.
- Log end-to-end traces: query → retrieval set → final answer → citations → permission checks.
- Add kill switches by capability (disable a connector, disable generation, limit tools) without redeploying.

Ship criteria (minimum bar)
- You can answer: who asked what, what sources were used, and why access was granted/denied.
- You can swap models without changing your retrieval pipeline.
- When the system is uncertain, it degrades to sources and questions—not confident nonsense.