Agentic RAG Launch Checklist (90-Day Workflow Plan) Goal Ship one production workflow (not a generic chatbot) that is grounded, measurable, and safe. 1) Pick the workflow + KPI (Week 1) - Choose ONE workflow with a clear owner (Support, SOC, Sales Ops, Finance Ops). - Define a single primary KPI (examples: handle-time reduction, higher first-contact resolution, faster onboarding). - Write explicit stop conditions: actions the agent must never take without approval (money movement, permission changes, outbound email to customers). 2) Build the gold dataset (Weeks 1–3) - Collect real historical cases (enough to represent common scenarios and edge cases). - For each case, store: user request, correct outcome, required citations (doc URLs/IDs), required tool actions (if any), and unacceptable outputs. - Redact PII/secrets and document the redaction approach. 3) Retrieval architecture (Weeks 2–5) - List corpora (policies, product docs, runbooks, tickets, knowledge base, CRM notes). - For each source, define a freshness SLA. - Use hybrid retrieval where exact identifiers matter (keyword + vectors + reranker). - Add metadata filters (tenant, product, region, version) to prevent cross-customer leakage. 4) Routing + memory (Weeks 4–7) - Add an intent classifier so requests route to the right sources and tools. - Implement scoped memory: - Session state: ephemeral and expires quickly. - Long-term profile: structured fields, user-consented, deletable. - Keep policies and docs in retrieval with citations—don’t store them as free-form memory. 5) Tool contracts + guardrails (Weeks 5–9) - For every tool: JSON schema, validation, clear error codes, retries, idempotency. - Add preview/dry-run mode for write actions. - Add approvals for high-risk actions; log approver, timestamp, and trace ID. - Implement post-conditions (verify the write changed the intended record). 6) Verification + refusal behavior (Weeks 6–10) - Citation verification: cited text must support the claim. - Add refusal rules when evidence is missing, stale, or conflicting. - Add clarifying-question behavior for ambiguous requests. 7) Evals + CI (Weeks 7–12) - Set thresholds per workflow for: citation validity, tool-call validity, policy violations, latency, and cost per successful resolution. - Run evals automatically on every prompt/index/tool change. - Add regression alerts and a rollback mechanism (model fallback, disable risky tools). 8) Launch + monitoring (Weeks 10–13) - Start with a limited cohort (internal users or a small traffic slice). - Monitor daily: escalation/hand-off rate, resolution rate, incident count, and cost per success. - Create an incident runbook: how to replay traces, isolate bad sources, and roll back changes. Definition of done - You can replay an agent run end-to-end (retrieval, model calls, tool calls). - You can answer procurement questions: retention, deletion, training use, and audit logs. - The workflow KPI improves without increasing high-severity incidents.