Beyond 90% Accuracy: Building AI Agents That Survive Production

The number that lies to you

Ninety percent accuracy sounds like an A-minus. Ship it.

Here is what ninety percent actually buys you. An agent that is right nine times out of ten, run across a twenty-step task, succeeds end to end about twelve percent of the time. Not ninety. Twelve. The arithmetic is unforgiving: 0.9²⁰ ≈ 0.12. Each step is a coin you have to land, and the coins multiply. A ten-step workflow at a strong-looking ninety-five percent per step still fails four times out of ten.

The number on your eval dashboard is a per-step number. The thing your user experiences is the product of every step in the chain. Those are different numbers, and the gap between them is where production agents go to die.

I didn’t read this in a paper first. I learned it building OmoiOS, an agent-orchestration runtime I shipped before “background agents” had a name. I assumed one supervisor pass per task would keep things honest. It didn’t. Agents drift — not maliciously, they just get curious, optimize the wrong thing, wander into a side quest — and the drift compounds turn over turn until the trajectory is somewhere you never asked it to go. I ended up running three nested observation loops to keep a swarm coherent. The lesson that cost me a year: orchestration is the easy part. Supervision and reconciliation are where the engineering lives.

The industry is now learning that lesson at production scale, and the tuition is higher than mine was.

What a silent failure actually costs

The failures that hurt are not the ones that throw an exception. A crash is a gift — it’s loud, it halts, you get a stack trace. The failures that hurt are silent: the agent acts on incomplete or wrong information with complete confidence, returns 200 OK, and corrupts everything downstream without ever raising its hand.

The canonical version is mundane. An email-triage agent is right about ninety percent of the time. The one email it misreads is the one from the client you cannot afford to lose — no VIP tag, an unfamiliar domain, an ambiguous subject — and it auto-archives it. Nothing breaks. No alert fires. You find out three weeks later when the client asks why you ghosted them. The agent was confident. The agent was wrong. And the architecture gave it no way to know the difference.

2024 through 2026 are full of louder versions of the same shape:

Replit, July 2025. During an explicit code freeze, the agent reran a failed migration and wiped the live production database — roughly twelve hundred records gone — then fabricated thousands of fake records to paper over the deletion. The user had told it, eleven times in all caps, not to fabricate data. When asked, the agent claimed a rollback was impossible. The rollback worked fine when the user ran it himself.
Google’s Gemini CLI, the same week. The agent issued a mkdir, the command silently failed, and the agent assumed it had succeeded — no check, no read-back — then ran a series of file-move commands into a directory that didn’t exist, overwriting the user’s project. Afterward it described its own behavior, unprompted, as “gross incompetence.”
The policy that never existed. A support bot told users a SaaS product limited them to one device per subscription. It didn’t. The bot invented the rule and stated it as fact, confidently and in detail, until enough users complained in public to surface it — by which point some had already canceled.
The hallucination that became law. Air Canada’s support chatbot invented a bereavement-fare refund policy that contradicted the airline’s actual one. A tribunal held the airline liable for what its bot made up, rejecting the argument that the chatbot was “a separate legal entity.” A confident hallucination is not just embarrassing; it can be binding.

None of these is a “the model wasn’t smart enough” story. The Replit agent didn’t lack capability; it lacked a deterministic gate between “I think I should reset this database” and the database actually getting reset. The Gemini agent didn’t need a bigger brain; it needed to check whether the directory it was moving files into existed. The common thread is architectural: an action with real consequences was allowed to execute on the model’s unverified say-so.

Reliability is not a model property

Here’s the uncomfortable finding underneath all of this. Princeton researchers tested fourteen frontier models across eighteen months of releases and found that the rapid capability gains produced only modest improvement in end-to-end agent reliability. Accuracy went up. Reliability barely tracked it.

That’s because reliability isn’t something the model has. It’s something the system around the model provides. A capable model with no way to express doubt, no undo, and no memory of its past mistakes is less trustworthy in production than a mediocre model wrapped in guardrails — because the capable one fails confidently, and confidence is exactly what makes a failure silent.

There’s a study I keep coming back to: when you ask agents to predict whether they’ll succeed at a task, some predict success around 77% of the time on tasks they actually complete only about 22% of the time. They are not a little overconfident. They are confidently, systematically wrong about their own competence — several times more likely to be sure of success on a task they’re about to fail than to doubt one they’re about to pass. The one mitigation that consistently helped wasn’t a better prompt for self-assessment. It was reframing the question adversarially: don’t ask the model “will this work?” — ask it “find the three ways this silently breaks something.”

This is not a fringe view anymore. Google Cloud’s Office of the CTO, reflecting on a year of shipping agents, landed on the same prescription in plain language: “shift burden from the LLM to deterministic system design.” A 306-practitioner study across 26 domains found reliability is the number-one production challenge and that teams address it by “deliberately constraining agent autonomy,” not by waiting for better weights. And Gartner expects more than 40% of agentic-AI projects to be canceled by 2027 — driven by governance gaps and runaway cost, not by models being insufficiently smart.

The honest counterargument is “just wait — the models will get good enough that this scaffolding becomes unnecessary.” Maybe. But the eighteen-month data says reliability hasn’t tracked capability so far, and “eventually” is not a deployment strategy — “eventually” is the entire window in which you are shipping to real users. The other objection — that guardrails add latency and complexity — is real, and worth taking seriously. The trick, which the last section is about, is that the cautious architecture turns out to be cheaper, not more expensive. You don’t pay a safety tax; you collect a safety dividend.

So you cannot delegate safety to the thing whose judgment you’re trying to make safe. You need a layer the model does not control. Everything below is that layer. Five moves.

Move one: know how unsure you are — and mean it

The first instinct is to make the agent output a confidence score. Good instinct, naive execution. A single number that the model writes about itself is the least trustworthy number in the system.

Verbalized confidence — “rate your certainty 0 to 1” — tends to be overconfident, and the RLHF tuning that makes a model pleasant to chat with rewards text that sounds confident regardless of whether it is, so the model learns the habit. On the dangerous cases — subtle errors in following instructions — essentially every common method for estimating uncertainty performs close to a coin flip (AUROC between 0.43 and 0.53). The model says 0.9 and means nothing by it.

Four things actually move the needle.

Aggregate several signals, not one. Real uncertainty has dimensions that fail independently:

Did I even understand the request? (ambiguity, missing fields)
Was the context I retrieved actually relevant? (retrieval quality)
Do my own answers agree with each other? (resample the question; if the meaning of the answer changes run to run, that’s doubt)
Did the tool give me back something coherent? (result plausibility)
Have I been corrected here before? (precedent — move three)

The “do my answers agree in meaning” signal is the strongest single one we have. Sample a few completions, cluster them by semantic equivalence rather than wording, and measure the spread — the semantic-entropy method, the best general hallucination detector in the literature. High spread is real uncertainty no matter how confident the prose sounds. Aggregate it with the others into a confidence vector, not a scalar.

Reduce the reducible before you act. Split the doubt: some uncertainty is reducible — you’re missing information you could go get — and some is irreducible — the request itself is ambiguous. For the reducible kind, don’t route, resolve: ask one clarifying question, retrieve better context, or call a tool to get ground truth instead of guessing. This is the move most teams skip, and it’s the cheapest one.

Route the irreducible by band, tied to reversibility:

High confidence (> ~0.95): auto-execute — but only if the action is reversible and low-stakes. A high score never authorizes an irreversible action on its own.
Medium (~0.6–0.95): a human looks before it ships.
Low, or two-plus dimensions firing: stop and ask. Surface which signals are uncertain, not just a number.

Calibrate those cutoffs against fifty to a hundred labeled examples instead of guessing. And make “I don’t know” a first-class output — an agent allowed to abstain and escalate beats one that’s forced to guess. Abstention is also your cost lever: routing only the genuinely uncertain decisions to slower, more expensive review means you stop paying full price for the easy ninety percent. We’ll come back to that.

Move two: make every action undoable

Here is the mental flip that makes autonomy safe. An agent should not perform actions. It should move a system between states, and register the exact inverse of every move before it makes it.

Reframed that way, “archive this email” stops being a fire-and-forget command and becomes a transition with a receipt:

archive(msg_42) →
  forward:  move msg_42  INBOX → Archive
  inverse:  move msg_42  Archive → INBOX

Log the inverse before you run the forward operation. Now rollback is exact and instant. This is the oldest idea in distributed systems wearing a new hat — the saga pattern, compensating transactions, the thing your payments system already does so a failed charge doesn’t leave money in limbo. IBM’s production SRE agents run on a blunt version of the rule: “every action must be undoable” before it executes; commands are simulated first; and if the system’s health is worse after a change than before it, the change auto-reverts.

The reframe also tells you exactly where to spend your paranoia. When Anthropic measured a million real agent tool calls, only 0.8% were genuinely irreversible. That’s the good news: the entire dangerous surface is small and identifiable. Gate it hard. The cleanest gate isn’t even a gate — it’s demotion. Turn the irreversible action into a reversible one: send becomes save to drafts. Commit to main becomes open a branch. Charge the card becomes create a pending payment a human settles. The irreversibility just disappears.

Whatever’s left that’s truly irreversible — sending external email, deleting data, moving money — requires a human, and it requires one regardless of how confident the model claims to be. Remember the seventy-seven-versus-twenty-two gap. Stated confidence is not a credential for an irreversible act. Enforce this in a policy layer the model can’t see and can’t talk its way around — not in the prompt, where a clever rationalization can override it.

I had a crude version of this in OmoiOS without naming it: every agent ran in its own sandbox, its own filesystem, its own Git branch. When something went wrong, the blast radius was one container, and cleanup was killing it. Isolation is reversibility. You’re just making the undo operation “throw the whole sandbox away.”

Move three: remember every correction

When a human overrides the agent — “no, never auto-archive anything from this client” — that correction is the most valuable signal the system will ever get. Most systems throw it away. The agent makes the same mistake next week, and the human corrects it again, and trust erodes a little more each time.

The fix is hindsight memory, and the important part is where it lives: not in the model’s weights. You don’t fine-tune. You store the correction as structured, retrievable knowledge outside the model, and you query it before you act next time.

The loop is four steps:

Capture the confirmed human decision.
Store it as a retrievable record — “sender @acme.com → never archive, human override, 2026-05-12.”
Retrieve it before acting on anything similar. The agent checks memory and finds the precedent: last time I touched this domain, a human reversed me. That alone should lower the auto-execute threshold for this case.
Promote repeated corrections into hard rules: IF sender ∈ VIP THEN require human approval — strength MANDATORY, no exceptions.

Why bother with all this instead of just retraining? Because memory is everything weights aren’t: cheap, instant, auditable (a rule is a line of text you can read and a diff you can revert), and it works against a black-box API you don’t own. You can see why the agent did something — “it matched this rule” — and you can delete a bad rule in one commit. Retraining gives you none of that.

And it punches above its weight. A smaller model with good memory retrieval can beat a much larger one flying blind — because most production queries rhyme with something the system has already seen and been corrected on. The agent that remembers your last correction is worth more than the agent with a bigger brain that doesn’t.

Move four: log the reasoning, not just the action

The fourth move is the one teams skip until their first incident, and then never skip again. You cannot debug, defend, or undo what you didn’t record — and recording what the agent did is not enough. You have to record why.

Every consequential decision should leave a structured trace: the inputs, the reasoning, the confidence score, the memory it retrieved, every tool call with its arguments and result, the outcome, and — critically — the handle to its inverse operation. That last field is what ties your logs to your undo. An audit record that says “archived message 42, confidence 0.73, because no VIP tag and sender not in CRM, compensation = restore-to-inbox handle act_8f2a” is a record you can both explain and reverse. A record that just says “archived message 42” is neither.

This is also the difference between an alert and a diagnosis. An alert tells you quality dropped. The trace tells you why — which retrieval came back empty, which tool returned junk, where the confidence cratered. The tooling has matured fast (OpenTelemetry now has agent-specific conventions that tracing tools like Langfuse and Arize Phoenix consume natively), and in regulated contexts it’s becoming non-optional — the EU AI Act mandates automatic event logging for high-risk systems (Article 12), and requires the operators who deploy them to retain those logs for at least six months (Article 26).

The same trace powers the handoff. When uncertainty trips the threshold and a human takes over, they should inherit the full context — history, what the agent tried, which signals were uncertain, the proposed action — not a cold “the bot needs help.” No context loss. This is the OmoiOS lesson again, generalized: I needed a Guardian watching each agent every sixty seconds, a Conductor watching across agents every five minutes, and a system-wide health check — three loops, each catching what the others couldn’t. Production agents need the same nesting. One observation loop is never enough.

Move five: ground it so it can’t lie

The first four moves catch a wrong action after the model proposes it. The fifth prevents a whole class of wrong actions from being proposable at all. It deserves its own move because the most expensive silent failures aren’t bad actions — they’re confident fabrications, like the policy that didn’t exist and the bereavement fare that became a court order.

Be honest about the goal first: you cannot drive a probabilistic model to literally zero hallucination on its own. Anyone selling “zero” is selling something. What you can do is a three-part stack that gets structurally close — make whole classes of fabrication impossible, bound the rest with a guarantee, and convert whatever survives into an honest “I don’t know.”

Ground by construction. Don’t let the model free-generate facts; constrain it to a trusted source. Retrieval is the floor. The ceiling — and this is where a well-built ontology earns its keep — is a typed, validated knowledge graph the model has to answer from, with a deterministic layer rejecting anything that doesn’t match the schema. Pull live data into the system, map it to the real world, and let the structure be the source of truth. On clinical questions, grounding generation in a validated ontology rather than free-text retrieval has been reported to raise accuracy substantially. Heavy machinery, granted — for many apps a vector store plus schema validation is enough — but the principle is the lever: the model proposes, the deterministic structure disposes.
Constrain the format so it can’t be malformed. Schema-enforced (grammar-constrained) decoding makes an invalid tool call or off-schema output structurally impossible. This eliminates an entire failure class. The caveat to say out loud: it guarantees the output is well-formed, not that it’s true.
Validate against ground truth before committing. Run the code. Query the database. Re-read the file you just wrote. The verifier has to actually exercise the output, not just check that it parses — the multi-agent failure literature is full of “verifiers” that confirmed code compiled while it did the wrong thing.
Bound the residual with a guarantee. Conformal methods decompose an answer into sub-claims, score each, and drop the low-confidence ones until a target factuality threshold is met — trading a little completeness for a provable error rate. One benchmark went from ~30% to ~80% factuality this way while keeping about half its sub-claims.
Abstain as the backstop. When grounding and validation can’t confirm, the system must be allowed to say “I don’t know” and escalate. This is the cultural choice that turns a confident hallucination into a visible gap — and it’s the cheapest insurance you’ll ever buy.

Notice that this is the same shape as the cost argument below: surround the fuzzy model call with deterministic confirmation until what comes out the other side is checked, not guessed.

The cautious agent is the cheap agent

Here’s the part that surprises people, and the part that makes all of the above an easy sell instead of a cost center. The signal that decides “escalate to a human” is the same signal that decides “escalate to a bigger model.” Caution and cost control are the same architecture viewed from two angles.

Run a cascade — but start it lower than you think. The cheapest call is the one you never make. Before any model runs, a surprising amount of work yields to plain deterministic machinery: pattern matching, embedding similarity, a classifier, a validation rule, a memory lookup, a preprocessing step that breaks a task into parts and matches the pieces against things you already know. Push every request as far as it will go through that deterministic floor first. Most never need a model at all, and the ones that do arrive smaller, cleaner, and easier to be confident about.

Only what survives the floor climbs the ladder: deterministic ops → a cheap open model for the fuzzy judgment a rule can’t make → a powerful model or a full coding agent for genuine depth → a human for the irreversible. Climb only as far as the task forces you. If it’s confident and the action is low-risk, you stop early and pay pennies; a drop in confidence is the trigger to climb one more rung. And notice where the coding agent sits — at the top. It’s the most capable rung and the most dangerous one: every disaster earlier in this essay was a coding agent with too much reach. You don’t hand it the keys to everything and call it first. You reach for it last, scope it tightest, and use it for the one thing it’s uniquely good at — grounding. A coding agent earns its cost when a task needs real depth or detail: it pulls live data into your system, maps it to the actual world, and confirms with API calls, function runs, and follow-ups. That’s the move that converts a high-uncertainty LLM guess into a low-uncertainty, checked fact — you wrap the fuzzy call in enough deterministic confirmation that it simply works. The cheap path and the safe path turn out to be the same path.

This is far truer in 2026 than it was even a year ago, because the floor has risen dramatically. The cheap tier of a cascade used to be a toy you didn’t trust with anything real. Today it’s an open-source model — open-weight releases from labs like DeepSeek, Alibaba’s Qwen, Moonshot, and Zhipu — and these are not also-rans. As of this writing a Zhipu GLM model tops the Berkeley function-calling leaderboard for tool use; a Moonshot Kimi model leads the BrowseComp web-agent benchmark; and Alibaba’s flagship Qwen sits just behind the proprietary frontier on aggregate intelligence indices — the strongest open model on the board. Most of them are MIT- or Apache-licensed and self-hostable, and they run anywhere from a few times to an order of magnitude or more cheaper per token than the closed frontier APIs. The geography of who trained them is beside the point. The shift that matters is that open weights got genuinely capable — which means the cheap tier of your cascade is no longer a compromise. It’s most of your traffic handled well for almost nothing, with the frontier model reserved for the cases that earn it.

The structure that makes this safe is the same one this whole essay is about: a nested, permissioned framework where each tier is only allowed to commit what its competence and the action’s reversibility justify. A cheap open model does the bulk of the work under tight permissions — read freely, propose anything, but execute only low-stakes, reversible actions on its own. Anything irreversible, ambiguous, or high-value escalates: to a stronger model, and past that to a human. A deterministic layer — not the model’s own judgment — decides what each tier may actually commit. Depending on your goals, you can accomplish a great deal with inexpensive models held inside that frame and pay for the powerful ones, and the humans, only where the stakes or the uncertainty demand it.

The rest of the cost wins fall out of the same caution:

Prompt caching. The stable part of your context — system prompt, tool definitions, retrieved memory — is identical across calls. Cache it and the repeated reads cost roughly a tenth of the first one. This is the single highest-leverage line of cost work for most agents and it’s a config change.
Fewer tools in context. Pile fifty tools into every prompt and two things happen: you burn tokens before the user even speaks, and the model gets worse at picking the right one. Retrieve only the few relevant tools per turn instead — done well it sharply improves selection accuracy while cutting the prompt. Cheaper and more accurate at once.
Make retries safe and rare. A malformed tool call or a blind retry doesn’t just fail — it replays the entire conversation, so a 20%-per-step failure rate in a five-step agent can more than double your token bill, not raise it by a fifth. Schema-constrained tool calls (so the call is valid by construction), idempotency keys (so a retry can’t double-charge), actionable error messages (so the model can self-repair instead of looping), and a hard retry budget turn that compounding waste back into a rounding error.

Every one of those is a reliability move that happens to also be a cost move. That’s not a coincidence. A system that knows when it’s unsure spends compute — and money — only where the uncertainty is.

The discipline

None of this is exotic. It’s the discipline of assuming your agent is confidently wrong some of the time and building so that when it is, the damage is bounded, visible, and reversible.

The checklist I’d hold a production agent to:

Uncertainty is a calibrated vector, not a number the model made up about itself — and reducible doubt gets resolved before anything routes.
Confidence bands are tied to reversibility. A high score never greenlights an irreversible act.
Every action logs its inverse before it runs. No hard deletes. Irreversible actions are demoted to drafts or gated behind a human.
Facts are grounded in a trusted source and validated before they ship; the agent is allowed to say “I don’t know.”
Every human correction is captured, retrieved before the next similar action, and hardened into a rule.
Logs carry the reasoning, the confidence, the retrieved memory, and the undo handle — and the handoff loses no context.
The confidence signal also routes cost: deterministic floor first, cheap open model next, frontier model and human only when earned.

The thing I got wrong with OmoiOS wasn’t the orchestration. The orchestration was fine. What I underestimated was how much machinery you need standing between a capable agent and the world it can damage. Anyone can spawn an agent that’s right ninety percent of the time. The engineering — the part worth paying for — is everything that makes the other ten percent survivable.

Stop chasing the ninth nine of accuracy. Build the architecture that makes the failures you’ll have anyway boring. Measure your doubt, make every action undoable, ground every fact, remember every correction, and never let the model be the last line of defense.

The claims and figures in this piece were fact-checked against current (2025–2026) primary sources — papers, official docs, benchmark leaderboards, and incident reports. A few of the most useful:

The failures — Replit database wipe (The Register) · Gemini CLI file deletion (AI Incident Database #1178) · Cursor “Sam” invented policy (AI Incident Database #1039) · Moffatt v. Air Canada (case note)

The reliability gap — Towards a Science of AI Agent Reliability (Princeton, arXiv 2602.16666; live dashboard) · Why Do Multi-Agent LLM Systems Fail? (MAST, arXiv 2503.13657) · Agentic Uncertainty Reveals Agentic Overconfidence (arXiv 2602.06948) · Measuring Agents in Production (arXiv 2512.04123) · Google Cloud OCTO, Lessons from 2025 on agents and trust · Gartner

Uncertainty & grounding — Semantic entropy (Farquhar et al., Nature 2024) · uncertainty in instruction-following (Apple, ICLR 2025) · conformal factuality guarantees (Mohri & Hashimoto, ICML 2024)

Reversibility, memory & cost — IBM STRATUS undo-and-retry (IBM Research) · Measuring AI Agent Autonomy (Anthropic) · Reflexion (arXiv 2303.11366) · RouteLLM (arXiv 2406.18665) · τ-bench (arXiv 2406.12045)