AI AUDIT TRAIL READINESS CHECKLIST (STARTUP EDITION)

Goal: Make every model-driven output explainable, reproducible (as much as practical), and defensible in an enterprise security/procurement review.

1) TRACEABILITY (REQUEST → OUTPUT)
- Every AI interaction has a unique trace_id exposed internally (and ideally in an admin UI).
- Logs link: user/workspace → UI action → gateway request → retrieval → tool calls → model response → post-processing → final output.
- You can export a trace bundle (JSON) for a single event without engineering intervention.

2) VERSIONING (BEHAVIOR CHANGE CONTROL)
- You record model provider + exact model name for each call.
- You version prompt templates, tool schemas, and policy rules.
- You can answer: “What changed between last week and now?” using release metadata.
- You have a rollback path for prompt/policy changes (even if model rollback isn’t possible).

3) PROVENANCE (WHERE DID THE ANSWER COME FROM?)
- For RAG: you log document IDs/URLs, chunk references, and retrieval query parameters.
- You enforce access control at retrieval time (no “index everything and hope”).
- Citations shown to users map to the same identifiers stored in logs.

4) POLICY + SAFETY CONTROLS (CONSISTENT ENFORCEMENT)
- You have a single policy layer (even if simple) that records allow/block decisions with a rule version.
- Tool calls are allowlisted per workspace/role; actions require explicit permission.
- You have “safe mode” toggles: disable tools, restrict outputs, or route to a safer configuration.

5) DATA GOVERNANCE (RETENTION, DELETION, TRAINING USE)
- You can state clearly whether customer data is used for training/fine-tuning and under what conditions.
- You have configurable retention for prompts/outputs (at least by workspace).
- You can perform deletion requests and produce a deletion record.

6) EVALUATION (REGRESSION TESTING FOR AI)
- You maintain a small, curated eval set tied to critical workflows.
- You run evals before shipping prompt/policy/model changes.
- You track failures by category (hallucination, policy violation, wrong tool action, etc.).

7) INCIDENT READINESS
- You have an on-call playbook: who responds, what to disable first, what to communicate.
- You can identify affected users/workspaces from logs.
- You can reproduce a reported issue using stored context (within your retention constraints).

If you can’t check these boxes, don’t add more “agent” autonomy. Build the audit trail first.