AI DATA EXHAUST REVIEW — 60-MINUTE OPERATOR CHECKLIST

Goal: For one production AI feature, identify every place sensitive context is copied (intentionally or not), then assign owners and controls.

1) Pick the feature boundary (5 minutes)
- Feature name + entrypoints (API routes, UI surfaces, background jobs)
- Model provider(s) used (OpenAI API, Anthropic API, Azure OpenAI Service, etc.)
- Retrieval sources (object storage, docs index, database tables, SaaS connectors)

2) Enumerate artifacts (10 minutes)
For each item below, mark YES/NO and where it’s stored:
- Prompts (system + user)
- Retrieved context snippets (RAG)
- Tool calls (inputs/outputs)
- Model outputs shown to users
- Safety/moderation decisions
- Evaluation datasets or replay logs
- Embeddings and vector indexes

3) Trace the telemetry path (10 minutes)
Write the actual path, not the intended one:
- App logs destination (CloudWatch/Stackdriver/other)
- Traces/metrics path (OpenTelemetry SDK → Collector → vendor)
- Any “export to bucket/warehouse” jobs (S3/GCS/Azure Storage, Snowflake, BigQuery)
- Ticketing/incident copies (Jira, Slack exports, pastebins—be honest)

4) Access reality check (10 minutes)
For each storage location, answer:
- Who can query it today (roles/groups)?
- Is access audited in that system?
- Is access reviewed on a schedule, or “set and forget”?
- Can contractors/support/vendors access it?

5) Retention & deletion (10 minutes)
- Retention period for each stream (raw prompts, raw context, sanitized aggregates)
- Deletion mechanism: can you delete by user/tenant/request ID?
- Are derived stores (embeddings, indexes, exports) included in deletion requests?

6) Controls to implement next (15 minutes)
Choose 3 changes you can ship without a re-architecture:
- Add redaction at the collection edge (OTel Collector or log forwarder)
- Replace raw payload logging with identifiers + gated retrieval
- Split debug vs audit streams (different retention, different access)
- Gate “debug mode” behind a feature flag with time limit + approver
- Stop one unnecessary export job and delete its historical backfill

Deliverable: a one-page register
- Feature: __________
- Owners: App ______ / Security ______ / Data ______
- Highest-risk artifact: __________
- Highest-risk storage location: __________
- Next 3 actions + dates: (1) ___ (2) ___ (3) ___