A lot of “AI product” work still looks like 2023: bolt a chat box onto an app, hope users rephrase until it works, then call it innovation. That era is ending, not because chat is useless, but because it’s a lazy interface for serious work.
The useful shift is quieter: products are becoming tool-calling systems. You expose capabilities (search, create, update, approve, refund, schedule, route, provision, deploy) as typed tools. An LLM becomes the router and planner; your product becomes the execution engine with guardrails. The experience doesn’t have to look like chat at all. It can be buttons, inline suggestions, background automation, or an API-to-API workflow.
If you’re a founder or operator: the competitive moat is moving from “we have an assistant” to “we have a reliable action layer.” The work is product design, not prompt writing.
The real product surface is no longer UI. It’s your tool catalog.
OpenAI made “function calling” mainstream; Anthropic formalized “tools”; Google pushed tool use into Gemini models; Microsoft anchored Copilot experiences on Microsoft Graph and connectors. The pattern is consistent: LLMs are better at deciding what to do than doing it directly. Your systems still need to do the work.
When a team ships a chat UI without a tool layer, they’re asking the model to “be the product.” That creates three problems immediately: (1) no deterministic path to action, (2) weak observability (you can’t measure what the assistant actually did), and (3) ugly safety tradeoffs (either you let it act freely or you neuter it).
The tool-calling approach is the opposite: the model never “acts” directly. It selects a tool; your system executes; you log inputs/outputs; you apply policy; you ask for confirmation when the action crosses a risk threshold.
Chat is a nice demo. Tool calling is an operating model.
Tool calling is a product decision disguised as architecture
Most teams frame tool calling as an engineering integration: define JSON schema, implement endpoints, ship. That’s backwards. The hard part is product: choosing which actions to make “AI-addressable,” how to represent them, and what the model is allowed to do without a human.
Design the tool catalog like it’s your public API—because it is
Even if you never expose tools externally, you’re creating an internal API that will be used by: (a) your own assistants, (b) user-authored automation, (c) agentic workflows, and (d) future partner integrations. If the tool catalog is messy, your entire AI layer becomes flaky.
Good tool catalogs share traits product teams already know from good APIs: stable names, small parameter surfaces, explicit permissions, predictable error modes, and “composability” (tools can be chained without weird side effects).
Stop giving models write access until you’ve earned it
Founders love the “it can take actions” pitch. Operators hate the incident review. The right progression is boring but effective: read-only tools → draft tools (create proposals) → write tools behind confirmation → write tools with policy-based auto-approval.
This mirrors what companies did with CI/CD permissions, database migrations, and financial controls: you build confidence via constraints and logs, not vibes.
Table 1: Comparison of real tool-calling implementations and what they’re actually good for
| Platform | Tool interface | Strength | Watch-outs |
|---|---|---|---|
| OpenAI API | Function calling / tools (JSON schema) | Broad ecosystem; common baseline for app patterns | You own policy, authZ, audit trails, and action gating |
| Anthropic API | Tools; strong emphasis on controlled tool use | Clean agent/tool separation; good for structured workflows | You still need deterministic system behavior around retries and partial failures |
| Google Gemini API / Vertex AI | Tool use + Google ecosystem connectors | Great fit for Google Cloud + Workspace-heavy stacks | Connector scope and enterprise policy decisions get complicated fast |
| Microsoft Copilot Studio | Actions/plugins; Graph and connector-centric | Enterprise distribution via Microsoft 365; strong admin controls | Product surface area is huge; easy to ship something incoherent |
| LangChain / LangGraph (open-source) | Tool abstraction + graph workflow orchestration | Fast prototyping; explicit multi-step execution graphs | You must impose software engineering discipline or it becomes a spaghetti agent |
What breaks in production: partial failure, retries, and ambiguous state
Chatbot failures are embarrassing. Tool-calling failures are expensive. The moment the model can create invoices, change entitlements, or trigger deployments, your error handling stops being a backend detail and becomes a product promise.
Make every tool idempotent or treat it like a financial transaction
LLMs retry. Orchestration layers retry. Networks retry. If your “create_refund” tool isn’t idempotent, you will eventually refund twice. This is old-school distributed systems reality wearing an AI costume.
Idempotency keys, request deduplication, and explicit state transitions aren’t optional. Stripe made idempotency a core primitive for a reason. If your product is becoming a tool platform, borrow the parts that prevented Stripe from melting down.
Don’t hide state. Make it visible and queryable.
The model needs to know what happened. Users need to know what happened. Support needs to know what happened. If the only record of an action is a blob of assistant text, you’ve built the least auditable system possible.
Expose an “activity ledger” for AI-initiated actions: who/what triggered it, which tools ran, inputs, outputs, and what approvals occurred. This is not a vanity feature; it’s your warranty.
Key Takeaway
If you can’t explain an AI-driven action as a tool call with a request ID, a permission check, and a logged result, you didn’t ship an AI feature—you shipped uncertainty.
The contrarian stance: “Agents” are a UI problem first
Silicon Valley loves the word “agent.” Most “agents” are just hidden UI flows. The product question is: how does a user steer and trust a system that can take multi-step actions?
Teams obsess over model choice and ignore interaction design. Then they wonder why adoption stalls after the first week.
Steerability beats autonomy
The best experiences won’t be fully autonomous. They’ll be interactive automation: the system drafts, proposes, pre-fills, and queues. Humans approve at the right moments. Over time, approvals become policies, and policies become defaults.
This is how GitHub Copilot became useful: not by shipping a robot programmer, but by putting suggestions into the exact place developers already work, with acceptance/rejection as the control surface.
Make “preview” the default state
If your assistant can change data, the default output should be a diff, not a paragraph. Think: “Here are the 12 records I’m going to update. Approve?” not “Done.”
Linear, Notion, Airtable, and Figma taught users to trust powerful tools because changes are visible, reversible, and inspectable. Apply the same idea to AI actions: previews, undo, and clear scoping.
Your “tool layer” is also your security layer
Once an LLM can call tools, prompt injection stops being a novelty and becomes a real attack path: malicious content tries to trick the model into calling privileged tools. This isn’t hypothetical; security teams at major vendors have been writing about it since the first wave of plugin ecosystems.
Tool calling forces a clean separation: content is untrusted; tool execution is privileged. Treat the model like an untrusted planner. Your app is the trusted executor.
Concrete controls that actually work
- Explicit scopes per tool: “read:tickets” is not “write:tickets.” Don’t pretend it’s the same permission.
- Per-tool allowlists of fields: let “update_customer” touch “notes” before it can touch “billing_email.”
- Confirmation thresholds: require human approval for high-impact actions (refunds, deletes, user role changes).
- Content isolation: separate retrieved text from tool instructions; never let retrieved content become executable directives.
- Audit logs that support forensics: store tool inputs/outputs and the identity that authorized them.
One practical pattern: policy as code in front of tools
If you already use policy engines (OPA / Open Policy Agent is a common choice) or RBAC/ABAC services, put them in front of tool execution. The LLM proposes; the policy decides. This is where enterprise buyers will eventually draw the line: not “does it use GPT-5 or Claude,” but “can our admins control it and prove what happened.”
# Example: tool request envelope you can log and evaluate with policy
{
"request_id": "req_01J...",
"actor": { "type": "user", "id": "u_123" },
"invoker": { "type": "model", "name": "assistant" },
"tool": "refund_payment",
"args": { "payment_id": "pay_456", "amount": "full" },
"context": { "workspace_id": "w_789", "ip": "203.0.113.10" }
}
Table 2: A tool-readiness checklist that maps directly to real production risks
| Tool readiness item | Why it exists | What “done” looks like | Where teams copy the pattern |
|---|---|---|---|
| Idempotency | Retries cause duplicate actions | Idempotency key + dedupe + safe retry semantics | Stripe API conventions |
| Permissions + scopes | Tool use is privilege escalation risk | RBAC/ABAC checks enforced server-side per tool | OAuth-style scopes; enterprise SaaS admin models |
| Human-in-the-loop gates | High-impact actions need explicit approval | Preview → approve → execute with clear diffs | GitHub PR review workflow |
| Observability + audit trail | You can’t debug or defend what you can’t trace | Request IDs, tool logs, stored inputs/outputs, actor identity | SRE tracing culture; SOC2 expectations |
| Rate limits + quotas | Agents can amplify load unexpectedly | Per-user and per-workspace quotas; backpressure and fallbacks | Public API management playbooks |
What to build next quarter: fewer prompts, more primitives
If you’re running product for a B2B SaaS app, your next quarter shouldn’t be “ship an agent.” It should be: build a small, high-quality tool catalog for the 5–10 actions that matter, and wrap it with approvals, logs, and idempotency. Then you can ship ten AI experiences on top of it without rebuilding the world each time.
A concrete sequence that won’t waste your time
- Pick one workflow with clear ROI and clear risk (ticket triage, CRM data cleanup, access provisioning). Avoid anything that touches money first.
- Define tools as product contracts: names, typed args, error modes, and what “success” means.
- Ship read-only + draft mode before write mode. Make the output a diff.
- Instrument an action ledger that support and security can use without engineers.
- Add policy gates and progressively relax them with admin-controlled rules.
The prediction to sit with: by the end of 2026, “AI features” will be evaluated the way we evaluate payments or auth—by reliability, auditability, and controls. The teams that win won’t be the ones with the fanciest model. They’ll be the ones that turned their product into a safe execution platform.
Ask yourself one question and be brutally honest: if an LLM tried to call your most important business actions 1,000 times in a day, would your system behave like a platform—or like a demo?
