MODEL SUPPLY CHAIN REVIEW (60 MIN) — TEMPLATE

Goal: In one hour, produce a concrete map of what models are in production (or about to be), what they can access, what they can do, and what can change without a code deploy.

ATTENDEES (keep it small)
- Tech lead for the AI surface area (required)
- Product owner (required)
- Security/AppSec partner (recommended)
- Data/Platform owner for retrieval sources (recommended)

PREWORK (10 minutes, optional)
- Bring the current architecture diagram if it exists.
- Bring a link to the repo/service that calls the model.
- Bring a list of tools/actions the assistant can trigger.

AGENDA
1) Inventory (15 min)
- List every model endpoint used (chat, embeddings, reranker, moderation). Include provider and where it’s called.
- List every “configurable” behavior surface: prompts, templates, routing rules, safety filters.
- List every place changes can happen: code deploy, feature flag, vendor-side update, admin console edit.

2) Data boundaries (15 min)
- What data can enter context? (retrieval sources, logs, tickets, docs, database fields)
- What is the access control mechanism for retrieval? (user/tenant scoping, ACL filters)
- What gets stored? (prompts, outputs, tool-call parameters) Where? For how long?
- Redaction: what sensitive fields are removed before prompts or logs?

3) Tool permissions (15 min)
- Enumerate tools the model can call.
- Classify tools into READ vs WRITE.
- For each WRITE tool: what approval gate exists outside the model? (UI confirm, workflow step, human review)
- Confirm a default-deny stance: what is OFF unless explicitly enabled?

4) Change management & rollbacks (15 min)
- What can you roll back in minutes? (model version, prompt version, tool permissions, retrieval config)
- Where are those switches? (feature flags, config files, gateway policy)
- What’s the “blast radius” if a model behaves badly? (which users/tenants, which actions)
- Who is on point during an incident? Name + escalation path.

OUTPUT ARTIFACTS (required)
- A one-page dependency map: Model calls, retrieval sources, tools/actions.
- A versioning plan: where prompts/policies live and how they’re reviewed.
- A rollback list: exact switches and owners.
- A short risk list: top 5 failure modes specific to this assistant.

LEADER’S PASS/FAIL QUESTIONS
- Can we trace any production output to a specific model ID + prompt version + retrieval sources?
- Can we show an audit log of tool calls with parameters and the user context?
- Can we disable WRITE tools without taking the whole system down?
- Can we withstand a provider model update without discovering it through user complaints?

If any answer is “no,” create an engineering ticket on the spot with an owner and a deadline. The point of this review is operational control, not documentation.