Retrieval-Native Agent Launch Checklist (2026)

Use this as a pre-launch gate for any “agent” feature that uses company data or can take actions.

1) Define the memory contract (write it down)
- What can be stored? (project decisions, runbooks, ticket summaries)
- What is explicitly forbidden? (credentials, secrets, private HR data, sensitive personal data unless required)
- Retention rules: how long does memory persist?
- Deletion rules: how can a user/admin delete personal or project memory?
- Provenance rules: what sources are allowed to inform answers?

2) Identify your memory stores (avoid “one bucket”)
- Working/session memory: ephemeral conversation state
- Project/team memory: curated docs, ADRs, runbooks
- Operational memory: tickets, incidents, deployments
- Personal memory: user preferences (only with consent)
For each store, list the system of record (Git, Jira/Linear, Confluence/Notion, PagerDuty, etc.).

3) Permission-aware retrieval (non-negotiable)
- Tie retrieval to your identity provider (Okta / Microsoft Entra ID) and app roles.
- Enforce access at query time, not only at indexing time.
- Verify tenant isolation with explicit tests (cross-tenant queries must return zero results).

4) Provenance and citations
- Store stable source identifiers (doc IDs, URLs, commit hashes, ticket IDs).
- Include citations in the UI and in logs.
- Version sources: if a policy doc changes, the agent must not cite an old version silently.

5) Retrieval quality controls
- Implement hybrid retrieval where it matters: lexical + vector.
- Add a re-ranking step for top candidates in high-stakes flows.
- Add freshness signals (timestamps) and deduplication.

6) Observability (so you can debug)
Log for every response:
- User identity/tenant (non-sensitive identifiers)
- Query text
- Retrieved items (IDs + snippets)
- Tool calls (inputs + outputs)
- Final answer text
- User feedback (thumbs up/down + optional reason)

7) Evals as CI gates
- Create an eval set from real user questions and known failure cases.
- Add regression tests for: relevance, correct citations, refusal behavior, tool safety.
- Gate releases on eval failures for high-risk capabilities.

8) Action safety (if the agent can change anything)
- Separate “draft” from “publish” tools.
- Require confirmation for destructive actions.
- Add rate limits and audit trails for all write operations.

9) Rollout plan
- Start read-only, then limited write, then broader permissions.
- Ship to internal users first with aggressive logging.
- Create an incident playbook: how to disable tools, freeze memory writes, and revert retrieval indexes.

If you can’t pass sections 3, 6, and 7, you’re not launching an operator. You’re launching a liability.