Agentic Search Readiness Checklist (Production)

Use this in design reviews before you expand sources or enable write actions.

1) Workflow scope
- Define one workflow with a clear “done” state (e.g., draft security questionnaire responses, prepare incident notes, triage support tickets).
- Write down what failure looks like (wrong doc cited, stale policy applied, action executed without approval).
- Decide what the system is allowed to do: read-only, suggest-only, or execute-with-approval.

2) Evidence contract (sources of truth)
- List authoritative systems for this workflow (e.g., Confluence runbooks, GitHub repo docs, ServiceNow knowledge base).
- Define precedence rules (if two sources conflict, which wins?).
- Require citations for every claim that could change a decision.
- Define staleness rules: what documents become “suspect” after updates elsewhere?

3) Permissions and identity
- Identify the identity boundary: SSO user, service account, or delegated OAuth per user.
- Confirm ACL propagation from each source (Google Drive/SharePoint/Confluence/Slack/Jira/Salesforce all differ).
- Enforce permissions at query time (not only at ingestion).
- Log access: user/role, filters applied, docs retrieved.

4) Indexing and lifecycle
- Choose an ingestion strategy per source: batch + incremental updates where available.
- Support deletions (tombstones) and fast removal from indexes.
- Track document metadata: source, path, owner, updated time, and permission identifiers.
- Define re-index triggers (source updates, permission changes, schema changes).

5) Retrieval quality and control
- Use hybrid retrieval where it helps (keyword + vector) for policy/spec domains.
- Add explicit filters (team, project, folder) instead of hoping similarity search “figures it out.”
- Prefer smaller, well-scoped corpora over “index everything.”
- Implement a “can’t answer” path when evidence is missing, unauthorized, or stale.

6) Tool calling and action safety
- Define tool schemas with strict typing and validation.
- Make write tools idempotent and support dry-run.
- Add approval gates for any state-changing action (PR merge, ticket closure, customer email).
- Scope credentials tightly (least privilege) and rotate keys.

7) Observability and evaluation
- Trace every run end-to-end: prompt, retrieved context IDs, tool calls, outputs.
- Create a golden set of real tasks and expected behaviors (including refusals).
- Run regression evals before shipping changes to prompts, retrievers, or source connectors.
- Add alerting on bad patterns: sudden citation drops, rising “unknown source,” repeated tool errors.

If you can’t answer “who accessed what, why, and what changed” from logs alone, you’re not ready to ship agentic search to serious customers.