Most “AI product” postmortems in 2025 sounded the same: hallucinations, retrieval that “didn’t work,” prompt regressions, tool calls gone rogue. The fix that followed was usually the same too: tweak the prompt, switch vector DBs, try a newer model.
That’s treating symptoms. The disease is that teams still ship prompts like they’re product code, while treating context like an accident. In 2026, that’s the dividing line: serious teams own a model context pipeline—a repeatable, testable system that decides what information a model is allowed to see, how it’s shaped, and how it’s audited.
OpenAI’s Assistants API, Anthropic’s tool use, Google’s Gemini models, Meta’s Llama ecosystem, and a pile of orchestration tools (LangChain, LlamaIndex, Haystack) made it easy to bolt LLMs onto apps. They also made it easy to ignore the hard part: your app’s truth. The model doesn’t know your business. Your pipeline does.
The contrarian take: “RAG” is not a feature. It’s an operating system concern.
Founders keep pitching “RAG-enabled” products as if retrieval is a differentiator. It isn’t. Retrieval is table stakes, and the naive version is actively dangerous: dumping top-k chunks into a prompt and praying the model behaves.
The more your product matters—support, finance ops, legal workflows, developer tooling—the less you can tolerate nondeterminism in what the model sees. Your customers don’t care that you used Pinecone or Weaviate or pgvector. They care that the answer matches policy, current state, and the transaction log.
So instead of “RAG,” think like this: you’re building a context compiler. Inputs: user intent, permissions, system state, enterprise data, policies. Output: a bounded, cited, auditable context package plus allowed actions.
Shipping prompts without a context pipeline is like shipping SQL without a schema: it works until it doesn’t, and then it fails in the worst possible ways.
What “context” really is (and why your architecture probably ignores half of it)
Context isn’t just documents. In most real systems, documents are the least reliable piece: they’re stale, inconsistently formatted, and full of edge cases.
The four context classes that matter
- Authoritative state: databases, ledgers, ticket status, inventory, account entitlements. If it can be queried, prefer it over prose.
- Policy: what the system is allowed to do—refund rules, approval chains, security constraints, compliance boundaries.
- History: conversations, prior actions, previous drafts, resolved incidents. Critical for continuity, risky for privacy.
- Knowledge: docs, wikis, PDFs, runbooks. Useful, but only when grounded and scoped.
Most teams build for class four and hand-wave the rest. That’s how you get an assistant that can quote the handbook but can’t tell whether an order shipped, whether a user is authorized, or whether the policy changed last week.
Tooling is converging. Your decisions are still hard.
The ecosystem has stabilized around a few practical building blocks:
Table 1: Comparison of common context-store approaches teams use with LLM apps
| Approach | Best for | Tradeoffs | Real examples |
|---|---|---|---|
| Postgres + pgvector | Teams already strong in SQL; unified transactional + vector storage | You own tuning, indexing strategy, and scaling patterns | PostgreSQL, pgvector |
| Managed vector database | Fast start; high-volume similarity search | Extra system to operate; cost and data gravity can surprise you | Pinecone, Weaviate Cloud, Milvus (managed offerings exist) |
| Search engine (hybrid lexical + vector) | Enterprise search with filters, facets, and relevance tuning | Schema and ranking work is real; not a “drop-in” RAG fix | Elasticsearch, OpenSearch |
| Data warehouse + embeddings | Analytics-heavy orgs; governance and lineage | Latency and query patterns can fight interactive workloads | Snowflake, BigQuery |
| Files + indexes (prototype mode) | Demos, internal tools, small corpora | Breaks under permissions, freshness, and audit needs | S3/GCS + DIY indexing |
The argument isn’t “use Postgres” or “use Pinecone.” It’s that storage is only one piece. The hard decisions live above it:
- How do you enforce permissions before retrieval, not after generation?
- How do you guarantee freshness for stateful questions (orders, outages, SLAs)?
- How do you prevent context poisoning (bad docs, prompt injection in retrieved text)?
- How do you test regressions when models and embeddings change?
The context pipeline: treat it like ETL, not prompt craft
A useful mental model is classic data engineering: ingest → normalize → enrich → index → serve → observe. The difference is that the “consumer” is a probabilistic model that will confidently fill gaps. That means your pipeline must be stricter than a BI dashboard.
A practical pipeline that holds up under real usage
- Ingest with provenance: every chunk knows its source URL/path, owner system, and last-updated timestamp.
- Normalize formats: HTML, PDF, Markdown, tickets, and spreadsheets become a consistent intermediate representation. Don’t feed the model raw sludge.
- Permission binding: attach ACLs at the smallest unit you will retrieve. If your permission model is only at “document level,” you will leak.
- Enrich: entity extraction, canonical IDs (customer_id, order_id), and link-outs to authoritative state.
- Index twice: lexical + vector. Keyword search catches exact terms, error codes, and part numbers that embeddings miss.
- Serve with budgets: token budgets, citation requirements, and “must-call-tool” rules for certain intents.
- Observe: log retrieved passages, tool calls, refusal reasons, and user feedback tied to the context package—not just the final answer.
Key Takeaway
If you can’t explain why a specific paragraph was retrieved and who was allowed to see it, you don’t have a context system—you have a demo.
“But we already have LangChain/LlamaIndex”
LangChain and LlamaIndex are useful, widely used, and real. They’re also not your architecture. They won’t magically solve governance, evaluation, or correctness. Treat them like libraries: helpful glue, not strategy.
# Minimal pattern: log the assembled context package, not just prompts
# Pseudocode (language-agnostic)
context = assemble_context(
user=user,
intent=intent,
permissions=get_acls(user),
state_queries=["order_status", "account_tier"],
retrieval_queries=[intent.query],
token_budget=6000
)
log_event("llm_context_package", {
"user_id": user.id,
"intent": intent.name,
"retrieved_sources": context.sources, # include doc IDs + timestamps
"tool_plan": context.tool_plan, # allowed tools + constraints
})
answer = model.generate(context=context)
Security and compliance: prompt injection is a retrieval problem first
The industry spent 2023–2025 arguing about jailbreaks and prompt injection. By 2026, the only prompt injection that matters is the one you retrieved from your own systems.
Attackers don’t need to “hack the model.” They just need to get malicious instructions into content your pipeline trusts: a public GitHub issue, a Zendesk ticket, a Confluence page, a shared Google Doc. If your retriever pulls it in, you invited the attacker into your system prompt.
Defenses that actually hold up
- Strict tool gating: tools are not “available.” They are allowed only when intent + policy says so.
- Content sanitization: strip or down-rank instruction-like text in retrieved documents (e.g., “ignore previous instructions”). Treat it as untrusted input.
- Source allowlists: for high-stakes flows, only retrieve from curated corpora, not the entire company drive.
- Mandatory citations: require the model to cite sources for claims, and fail closed when citations are missing.
- State over prose: if a question can be answered via an API call, do it. Don’t let a doc override reality.
Evaluation: stop scoring “answers,” start scoring context packages
LLM evaluation tools exploded: prompt regression tests, golden datasets, and “LLM-as-judge” scoring. Useful, but incomplete. If you only score final answers, you’ll miss the root cause of failures: wrong or missing context.
What you want is a second set of tests aimed at the pipeline itself: retrieval accuracy, permission correctness, freshness, citation coverage, and tool-plan quality.
Table 2: Context-pipeline checks you can run continuously (CI + production monitoring)
| Check | What it verifies | How to implement (publicly known patterns/tools) | Failure mode it catches |
|---|---|---|---|
| Permission retrieval test | Docs/chunks returned match user ACLs | Synthetic users + fixtures; assert retrieved IDs subset of allowed set | Data leakage across tenants or roles |
| Freshness guardrail | Answers depend on current state, not stale text | Force tool calls for intents (e.g., order status) and reject doc-only plans | Confident but outdated responses |
| Citation coverage | Claims map to sources | Require structured output with citations; validate each claim has a source ID | Hallucinated facts that look plausible |
| Retrieval quality set | Retriever pulls the right passages for known queries | Curate a query→source set; test lexical + vector configs on every change | Silent regressions from re-embedding or re-indexing |
| Tool-plan policy test | Model chooses allowed tools and respects constraints | Contract tests on tool schema + policy layer; replay traces | Unauthorized actions, runaway tool loops |
Notice what’s missing: model brand. These checks survive model churn. That’s the point. Founders still get stuck in “which model should we use?” as if that’s strategy. The teams that win can swap OpenAI, Anthropic, Google, or open-weight models (like Meta’s Llama family) because their product quality lives in the pipeline.
A sharp prediction for 2026: context engineers will be more valuable than prompt engineers
“Prompt engineer” was a 2023 job title. By 2026, it’s mostly a skill inside product and engineering. The dedicated role that sticks is closer to data/platform/security: people who understand schemas, permissions, lineage, and evaluation.
Two things will make this obvious:
- Regulated buyers will force it. If you sell into banks, healthcare, or government, “trust us” won’t cut it. You’ll need audit trails: what was retrieved, what was generated, what was executed.
- Agentic workflows will raise the blast radius. Tool use is real: models can draft emails, file tickets, run queries, update CRM objects. That only works if the plan is policy-bound and observable.
If you’re building a product with LLMs, take one concrete action this week: pick your highest-stakes user flow and write down, explicitly, what sources are allowed, what tools are allowed, and what must be queried live. If you can’t write it in a page, your system is already out of control. Fix that first—then argue about models.
The question worth sitting with: if your model vendor disappeared tomorrow, would your product still know what’s true?
