Most AI teams still treat models like products. They aren’t. Models are dependencies—volatile ones—with behaviors you only understand after they break in production.
The mistake shows up in the same place every time: a team ships an “agent,” ties it to revenue workflows, then discovers they can’t answer basic operator questions. What changed? Why did this response happen? Which tool call caused the failure? Can we reproduce it? Can we roll it back without turning off everything?
2026’s winners won’t be the teams that found the perfect model. They’ll be the teams that built testable AI systems: evaluation-first pipelines, versioned prompts and tools, auditable traces, and release engineering that looks closer to SRE than “prompt magic.” If that sounds unsexy, good. Unsexy wins.
Key Takeaway
If you can’t write an eval for it, you don’t understand it. Treat every model change, prompt change, retrieval change, and tool change like a production deploy—with gates, traces, and rollback.
“Agents” didn’t fail. Un-instrumented software failed.
There’s been plenty of discourse about whether “AI agents” are real or hype. Wrong argument. The only question that matters to founders and operators is whether the system can be operated—observed, tested, upgraded, and reverted—without heroics.
We already have a working mental model for this: distributed systems. When you take a model (OpenAI, Anthropic, Google, open-source on vLLM), wrap it in orchestration (LangChain, LlamaIndex, or internal code), add tools (Stripe, Salesforce, GitHub, internal APIs), and throw retrieval into the mix (Pinecone, Weaviate, Elasticsearch/OpenSearch, Postgres pgvector), you’ve built a distributed system with nondeterministic components.
And like every distributed system, the failures are rarely “the model.” They’re integration failures: stale indexes, brittle tool schemas, permission drift, silently truncated context, prompt regressions, or a vendor model update that changes behavior in a way you didn’t test for. If you’re not collecting traces and running evals against production-like traffic, you’re shipping hope.
The contrarian take: fine-tuning is being over-prescribed
Fine-tuning is a tool. It’s not the first tool. The industry’s default to “let’s fine-tune” often masks a more basic failure: the team doesn’t have evals, doesn’t have ground truth, and doesn’t know which part of the system is responsible for the behavior.
In practice, a lot of what teams call “model quality problems” are actually:
- Specification problems: you haven’t written down what “good” looks like in a testable way.
- Retrieval problems: the right data isn’t being pulled (or it’s being pulled but poorly packed into context).
- Tooling problems: tool schemas are ambiguous; errors aren’t surfaced; timeouts and retries are wrong.
- Workflow problems: you’re asking a single step to do what should be a staged pipeline with explicit checkpoints.
- Governance problems: nobody can reproduce outputs because prompts, tools, and data snapshots aren’t versioned.
Fine-tuning can help when you need consistent style, domain phrasing, or structured outputs. It can also help reduce prompt length and cost. But if you haven’t stabilized your retrieval and tool calling—and you don’t have evals that catch regressions—fine-tuning just makes your system more confidently wrong.
“You can’t improve what you can’t measure.”
That line is usually attributed to Peter Drucker, though the attribution is debated. The sentiment is still correct for AI systems: teams that skip measurement ship vibes, not software.
What “testable” means in real AI stacks
Testable doesn’t mean “we ran a demo.” It means you can run the system against a fixed dataset of scenarios, score it, inspect failures, and know what changed. It means your agent has the same basic properties you expect from other production services: observability, change control, and rollback.
Start with evals that reflect the workflow, not the model
Engineering teams get trapped because they evaluate a chat completion in isolation. That’s not what’s running in production. Production is retrieval + tools + policies + retries + timeouts + post-processing. Your eval harness should run the whole thing end-to-end.
Concrete, publicly-available building blocks exist:
- OpenAI Evals (open-source) for defining and running evaluation tasks.
- LangSmith (LangChain) for tracing, dataset management, and evaluation workflows.
- Weights & Biases (W&B) for experiment tracking, including LLM evaluations and prompts as artifacts.
- Arize Phoenix (open-source) for LLM observability and evaluation.
- Ragas (open-source) for RAG evaluation patterns (use carefully; automated metrics can mislead if you don’t validate with humans).
The point isn’t which tool you pick. The point is that evals become a gate, not a slide deck.
Version everything that changes behavior
Teams version code but forget the rest: prompts, system instructions, tool schemas, retrieval settings, chunking, reranking, and even the data snapshot used to build the index. If your “same question” yields different answers after a deploy, you need to know whether the change came from a model version, a prompt tweak, a different top-k, or a new document in the corpus.
At minimum, store these as deployable artifacts:
- Prompt templates + system messages
- Tool schemas and tool selection logic
- Retriever configuration (chunk size, overlap, embedding model, top-k, filters)
- Index build inputs (document versions, timestamps)
- Model identifiers (provider + model name/version)
The 2026 tool reality: don’t pick “a platform,” pick an operating model
Founders waste cycles arguing “LangChain vs LlamaIndex” or “vendor A vs vendor B.” That’s a 2023 conversation. In 2026, the real decision is whether your team is building an operator-friendly system: traces you can query, evals you can run on demand, and artifacts you can roll back.
Table 1: Common evaluation + observability options (what they’re actually good at)
| Tool | Type | Strength | Watch-outs |
|---|---|---|---|
| LangSmith | Tracing + datasets + evals | Tight loop for debugging chains/agents; good UX for traces | Best fit if you’re already in LangChain land; avoid coupling your whole architecture to one SDK |
| Arize Phoenix | Open-source observability/evals | Self-hostable; integrates with OpenTelemetry patterns | You still need to define what “good” means; tooling doesn’t replace eval design |
| Weights & Biases (W&B) | Experiment tracking + artifacts | Strong versioning discipline; works across ML + LLM workflows | Easy to drown in runs without a clear eval taxonomy and ownership |
| OpenAI Evals | Open-source eval harness | Simple mental model; good for repeatable checks | You’ll likely extend it for full system tests (RAG + tools), not just single calls |
| OpenTelemetry (OTel) | Instrumentation standard | Vendor-neutral tracing/metrics; fits existing SRE practice | Requires engineering effort; you must define semantic conventions for LLM events |
Here’s the position teams should adopt: your LLM stack should emit traces like any other production service. If your AI vendor gives you a nice dashboard, fine. But don’t mistake a vendor dashboard for an operating model. You want portable telemetry and portable datasets.
How to ship AI like a serious production system (without slowing down)
“Move fast” doesn’t mean “skip controls.” The teams that ship fastest are the ones with tight feedback loops: small diffs, automatic checks, and fast rollbacks.
A release pipeline that matches AI’s failure modes
This is the sequence that works in practice because it maps to where AI systems actually break:
- Build a scenario dataset from real tickets, failed runs, and high-value workflows. Label what success means (even if it’s just “acceptable / unacceptable” at first).
- Run end-to-end evals that exercise retrieval, tool calls, and post-processing, not just the base model response.
- Trace everything in staging with production-like auth, rate limits, and timeouts. Make tool errors visible.
- Canary the change to a small traffic slice or internal users. Compare to baseline runs.
- Promote with a rollback plan: keep last-known-good prompt/tool/retrieval configs ready to revert without a code deploy.
Notice what’s missing: “argue about the perfect model.” Model choice matters, but operational discipline matters more—and it’s the part most teams avoid because it forces clarity.
One practical trace format teams can adopt immediately
If you’re not ready to standardize on a full observability product, you can still structure traces. This minimal JSON shape is enough to debug most agent failures:
{
"request_id": "...",
"model": "provider/model-name",
"prompt_version": "...",
"retrieval": {
"index_version": "...",
"top_k": 8,
"docs": [{"id": "...", "score": "..."}]
},
"tool_calls": [
{"name": "stripe.create_refund", "status": "ok", "latency_ms": "..."},
{"name": "salesforce.update_case", "status": "error", "error": "..."}
],
"output": {"final": "..."}
}No magic. Just enough structure that a human can answer: what did it see, what did it do, what failed, and which version produced it.
RAG isn’t a feature. It’s an operational liability unless you treat it like data engineering.
Retrieval-augmented generation (RAG) became the default pattern because it’s often the fastest path to domain usefulness. But most RAG systems are built like a hack: dump PDFs into a vector store, chunk them, and hope similarity search does the rest. That works until it doesn’t—then you’re in a reliability hole that looks suspiciously like traditional data quality.
The two RAG failures that keep recurring
Staleness: your knowledge changes but your index doesn’t. If you don’t have an ingestion schedule, document versioning, and deletion semantics, you’re training the model to lie with outdated context.
Context packing: even if you retrieve the right chunks, you can still feed them in a way that confuses the model—too long, too redundant, missing the key paragraph, or mixing conflicting policies.
Table 2: A concrete AI system checklist (versioning + tests + ops)
| System Surface | What to Version | Minimum Test | Operator Signal |
|---|---|---|---|
| Prompts | Templates, system messages, tool instructions | Golden set of scenarios; regression checks on forbidden outputs | Diff view + prompt version in every trace |
| Models | Provider + model name/version; decoding settings | Side-by-side eval against last-known-good | Alert on behavior drift for high-value intents |
| Retrieval | Embedding model, chunking, filters, index build snapshot | Question-to-source tests: can it retrieve required docs? | Dashboards for top missing docs + stale docs |
| Tools | Schemas, permissions, rate limits, error handling | Contract tests + simulated tool failures/timeouts | Tool error rate, retries, latency per tool |
| Policies | Safety rules, redaction, allow/deny lists | Adversarial prompts; PII leakage checks | Audit logs for sensitive actions + blocked outputs |
If you’re building RAG without an ingestion and deletion plan, you’re not building an AI feature. You’re building a shadow knowledge base with no owner.
A prediction worth building around: AI vendors will compete on operability, not just intelligence
Model labs will keep shipping stronger models—OpenAI, Anthropic, Google, and the open ecosystem around Meta’s Llama family have made sure of that. But as model quality rises, differentiation shifts to something founders can actually monetize: uptime, controllability, auditability, and debuggability.
That means your defensibility won’t come from having picked “the best model.” Your defensibility comes from having a system that can safely ride model churn: swap providers, update versions, add tools, change retrieval, and still meet SLAs because your evals and traces catch regressions early.
Next action: pick one revenue-critical workflow and do the unglamorous work this week. Create a 50-scenario dataset, run it end-to-end, and wire a CI gate that blocks deploys when it regresses. If that feels heavy, good. That’s what turning AI into software actually requires.
One question to sit with: if your primary model provider changed output behavior tomorrow, could you prove—within an hour—whether your product got better or worse?
