AI Agent Reliability in Production: Error Handling, Retries, and the Math of Failure
There’s a number that changes how you think about agent deployment: GPT-4-based agents score around 60% on standard benchmarks using pass@1 — success at least once. When the evaluation switches to pass@8 — requiring consistent success across eight independent runs — the number drops to 25%. The same agent. The same tasks. A completely different picture of what deployment actually looks like.
This isn’t a benchmark artifact. It’s a measurement of how much trust you should have in a demo versus how much you should have in a deployment.
Part 1: The Math That Kills Autonomous Agents
Error Compounding: Why Long Tasks Fail Predictably
Suppose an agent has a 99% accuracy rate per step. It does the right thing 99 times out of 100. That sounds excellent. Now ask it to complete a 100-step task:
P(success over 100 steps) = 0.99^100 = 0.366
# Even at 99% per-step accuracy: 36.6% success on 100-step tasks
P(success) = 0.95^100 = 0.006 # 95% per-step → 0.6% success
P(success) = 0.90^100 = 0.000027 # 90% per-step → 0.003% success
P(success) = 0.99^10 = 0.904 # 99% per-step → 90.4% success, 10 steps| Per-Step Accuracy | 10-Step Task | 50-Step Task | 100-Step Task |
|---|---|---|---|
| 99% | 90.4% | 60.5% | 36.6% |
| 97% | 74.0% | 22.1% | 4.9% |
| 95% | 59.9% | 7.7% | 0.6% |
| 90% | 34.9% | 0.5% | 0.003% |
Real agents on realistic tasks make mistakes at somewhere between 5% and 20% per step, depending on task complexity. Which means that for tasks over ~20 steps, failure is not an edge case — it’s the expected outcome.
This is not a reflection of the current state of model capability. It’s a mathematical property of sequential processes. It will not be solved by making models smarter. An agent with 99.9% per-step accuracy still fails 9.5% of the time on a 100-step task. The only escape from error compounding is checkpointing — breaking the multiplication chain.
The Spiral of Hallucination
The mathematical failure mode has a behavioral signature researchers have named the “Spiral of Hallucination”:
- The agent makes a minor error in step 3 — misidentifying a constraint, mis-reading an output, making an incorrect assumption.
- Subsequent reasoning builds on this error. Each step takes the wrong premise as given.
- By step 15, the agent is operating from an internally consistent but factually wrong model of the situation.
- Corrections become increasingly difficult because the agent’s context is now saturated with correlated wrong conclusions.
The spiral is worsened by a counterintuitive property of chain-of-thought reasoning models: they can hallucinate more on complex tasks, not less. Because they build elaborate reasoning chains from their premises, a single wrong premise early on produces a cascade of confident, internally-consistent, wrong conclusions.
Why Larger Context Windows Don’t Fix This
A common intuition: if agents fail because they lose track of information, give them bigger context windows. GPT-4 supports 128k tokens. Claude supports 200k. That should help, right?
The evidence says no. The “Lost in the Middle” research (Liu et al., 2023, replicated on 18 frontier models) found a consistent U-shaped attention curve. Models reliably attend to information at the beginning and end of long contexts. Information in the middle gets systematically underweighted — not ignored, but consistently under-utilized.
Longer context windows don’t eliminate the problem; they increase the amount of information that falls into the underweighted middle. Loading 100k tokens of context is not the same as having 100k tokens of reliable working memory.
Context windows prevent forgetting, but error compounding is a cumulative accuracy problem, not a memory problem. At 95% per-step accuracy, 100 steps gives 0.6% success regardless of context window size.
Part 2: The Reliability Gap Research Doesn’t Want You to Know
Capability vs. Reliability Are Diverging, Not Converging
The Princeton paper Towards a Science of AI Agent Reliability (arXiv:2602.16666, February 2026) ran 14 agentic models with K=5 repeated runs per task across two benchmarks. The top-line finding across 18 months of model releases: “reliability gains lag noticeably behind capability progress.”
Despite steady accuracy improvements, reliability has shown only modest overall improvement. Smarter agents are not more reliable agents — at least not by the metrics that matter for deployment.
Pass@1 measures what your agent can do. Pass@8 measures what your agent will do. For users in production, only the second number is real. The first one is a demo.
The Four Dimensions Standard Benchmarks Don’t Measure
Consistency — Does the agent produce equivalent outputs when given semantically equivalent inputs? The paper used five paraphrased versions of each prompt and measured whether the agent behaved the same across all of them. Finding: consistency scores are uniformly low across all 14 models, including frontier models.
Robustness — Does the agent behave appropriately when given inputs with injected faults? Most models degraded significantly — and not gracefully. Graceful degradation (failing safely, signaling uncertainty) is different from fragility (failing silently, continuing with garbage input). The models mostly showed fragility.
Predictability — Is the agent calibrated? Does it know when it’s likely to be wrong? Discrimination on GAIA has “mostly worsened” across 18 months of model releases. You can’t predict which model will be reliable for your specific task without testing it directly.
Safety — When the agent fails, does it fail safely? High-severity violations are rare, but financial accuracy violations are the most prevalent failure mode — which is exactly the type of failure that causes real-world harm.
The Benchmark Problem Within the Benchmark Problem
The Princeton researchers discovered that 24 out of 50 original τ-bench tasks contained errors. Nearly half. The benchmark was measuring agent performance against tasks that were partially broken.
The practical implication: when you test your agent and it “passes,” you might be checking it against a broken standard. The GAIA benchmark human-AI performance gap stands at 77% (humans reliably succeed where agents don’t), suggesting real-world performance gaps are much larger than lab conditions imply.
A 2025 survey of 306 AI agent practitioners found reliability issues were the single biggest barrier to enterprise adoption — not capability, not cost, not data access.
Part 3: The Hidden Cost of Retries
Why Agent Retries Are Nothing Like Traditional Software Retries
Every agent builder hits the same moment. An API call fails. The natural instinct is to wrap it in a retry loop with exponential backoff and move on.
For traditional software, this works. A REST API retrying a database query has a fixed, predictable cost: one more network round-trip, maybe a few milliseconds of compute.
For LLM-based agents, retries are anything but free. Each retry carries at least three hidden costs:
Cost 1: Token Multiplication When an agent retries a tool call, it doesn’t just re-send the tool invocation. It re-sends the entire conversation context up to that point. Every message, every previous tool result, every system prompt — all of it gets re-tokenized and billed as input tokens on the retry.
Suppose your agent is 15 steps into a task. The accumulated context is around 8,000 tokens. A tool call fails. The retry sends those 8,000 input tokens again, plus the new attempt. With a 5-10% failure rate — realistic for production agents making external API calls — you’re looking at a 10-20% cost overhead just from retries.
Cost 2: Context Pollution When a tool call fails and gets retried, the failure — including the error message, the agent’s reasoning about why it failed, and the decision to retry — stays in the conversation context. The agent now has to reason about a longer, noisier context that includes a failure pathway it shouldn’t be thinking about anymore.
After a failed tool call, agents sometimes become “cautious” about using that tool again, even after the retry succeeds. They add unnecessary validation steps, or try to work around a tool they just used successfully. The failure is in the context window, and the model attends to it.
Cost 3: Cascade Failures Agent retries can cascade in ways that are hard to predict and hard to debug. Consider: Agent calls Tool A, which depends on Tool B’s output. Tool B fails, so the agent retries Tool B. The retry succeeds, but returns slightly different data than the original attempt would have (maybe a different timestamp, or data that changed during the retry window). Tool A receives this slightly-different input. It doesn’t fail — it just produces a subtly wrong result that propagates through the rest of the session.
In agent loops that run autonomously, cascade failures from retries can compound across sessions. A retry in session N produces a slightly wrong file. Session N+1 reads that file as ground truth. By session N+3, the original retry is invisible in the logs, and you’re debugging a problem that seems to have appeared from nowhere.
Part 4: Architectural Responses That Actually Work
1. Checkpointing: Break the Sequential Chain
The most direct response to error compounding: break the sequential chain. After every 5-10 steps, the agent should explicitly verify its current state against the expected state at that point:
- What was the goal of this task?
- What state did I expect to be in at step 10?
- What state am I actually in?
- Is this approach still valid?
A checkpoint that catches a wrong premise at step 5 prevents the Spiral of Hallucination from developing. The cost is one extra LLM call per 5-10 steps. The payoff is orders of magnitude better success rates on long-horizon tasks.
2. Classify Errors Before Retrying
Not all errors are retriable. Classify the error first, then decide whether to retry, fail gracefully, or try an alternative approach:
if status == 429:
retry with exponential backoff + jitter
elif status >= 500:
retry once, then fail with context
elif status == 400 or status == 401:
do not retry — the request is wrong
else:
log and escalateA 429 (rate limit) will probably resolve with backoff. A 400 (bad request) won’t — retrying it is pure waste. 401 is not retryable. Permanent auth failures should immediately escalate to the operator rather than consuming retry budget.
3. Circuit Breakers, Not Just Retries
After 2-3 failures of the same tool in a session, stop trying. A circuit breaker pattern — where you track failure rates per tool and stop calling tools that are consistently failing — prevents the agent from burning through your token budget on a service that’s genuinely down.
The per-session retry budget model: not “max 3 retries per call” (the standard approach), but “max 10 retries per session, across all calls.” This forces the agent to be economical about which failures are worth retrying.
4. Strip Failed Attempts From Context
If a tool call fails and you’re going to retry, remove the failed attempt from the conversation context before retrying. This prevents context pollution. Not all agent frameworks make this easy, but the ones that let you manage message history explicitly have a significant advantage here.
5. Write Intermediate State to Disk
If a tool call succeeds, write the result to a file immediately. If the next step fails and needs to be retried, you can reload the previous result from disk instead of re-deriving it. This breaks the cascade chain — retries don’t propagate backwards through the dependency graph.
6. Bounded Autonomy by Design
Organizations that deployed “Level 5” autonomous agents (fully unsupervised) had substantially worse outcomes than those that deployed “Level 3-4” agents (autonomous within scope, with defined escalation paths).
The winning architecture: the agent knows its operational domain precisely. When a task falls within the domain, it executes fully autonomously. When it falls outside — or when uncertainty exceeds a threshold — it escalates to a human rather than guessing. Bounded autonomy with principled escalation is the architecture that survives in production.
7. Verifiable Rewards Over Self-Assessment
The Agent-RLVR paper (2025) took Qwen-2.5-72B-Instruct from 9.4% to 22.4% pass@1 on SWE-bench Verified through reinforcement learning with verifiable rewards — not by changing the base model, but by using unit test pass/fail as a training signal. The mechanism: attempt → test → binary result → guidance → reattempt → updated policy. No self-evaluation. No narrative reflection. Just a clean, unambiguous signal.
This is why software engineering is the domain where agents have improved fastest. Tests provide verifiable rewards. The practical upshot: if you can make your agent’s success criteria verifiable, you can build a reliable improvement loop. If you can’t, you need to be much more conservative about how much autonomy you grant.
What Actually Improves Reliability
The Princeton paper identifies calibrated confidence as the most underrated reliability property. An agent that says “I’m not sure about this” when it isn’t sure, and proceeds confidently only when it actually knows — this agent is more deployable than an overconfident agent with better raw accuracy. The miscalibrated confident agent fails without warning. The calibrated uncertain agent fails visibly, and the human operator can intervene.
The practical checklist:
- Test pass@k, not just pass@1. Run the same scenarios multiple times. If your agent’s success rate drops significantly from pass@1 to pass@5, your benchmark number is not your deployment number.
- Measure consistency directly. Generate semantically equivalent versions of your test inputs and check whether behavior is equivalent.
- Require explicit uncertainty signals. Agents that say “I’m not confident about this” are more trustworthy than agents that never express uncertainty. Design your evaluation to reward calibration, not just accuracy.
- Measure behavior, not just outcomes. An agent that gets the right answer via an unreliable path is not the same as an agent that gets the right answer via a reliable one. Track the path, not just the result.
- Implement mid-task hypothesis checks. After completing what feels like the “act” phase, explicitly re-read the hypothesis from the start and ask: am I still on track? Did reality diverge from prediction?
- Objective verification over self-assessment. After any significant action, verify with an actual test: is the service responding? Does the page render correctly? Did the database record appear? Not “I think I did this right” but “the system confirms this worked.”
Frequently Asked Questions
What is error compounding in AI agents? Error compounding is the multiplicative accumulation of small mistakes across an agent’s sequential steps. Each step’s output becomes the next step’s input. A 98% per-step accuracy rate becomes only 13% success after 100 steps (0.98^100 = 0.13). Unlike human errors that tend to be caught, agent errors compound silently: the agent does not know it made a mistake, and subsequent steps reason from the wrong premise as if it were correct.
What success rate can I realistically expect from AI agents on long tasks? METR research shows agents complete tasks requiring approximately 1 hour of human work at roughly 50% success rate. On challenging real-world software tasks (SWE-Bench), success rates are 13-38%. On GAIA Level 3, even frontier models achieve under 20%. A task requiring 50 correct sequential decisions at 95% per-step accuracy has a 7.7% success rate.
What is the Spiral of Hallucination in autonomous AI agents? The Spiral of Hallucination is when a wrong early premise gets embedded into the agent’s working state and every subsequent step reasons from that wrong premise. The error compounds silently: each step produces a locally coherent output given its incorrect input, so there are no obvious failure signals. The agent reaches the end of a 50-step task with high internal confidence, having built an elaborate but completely wrong conclusion on a foundation error it never detected.
How do you stop error compounding in autonomous AI systems? Four architectural approaches: (1) Mid-task hypothesis checks — re-read the original goal at the midpoint of long tasks and verify reality has not diverged. (2) Bounded scope — define a narrow specific goal before starting and refuse scope expansion mid-task. (3) Objective verification over self-assessment — verify with a real signal such as an HTTP status code or exit code, not internal evaluation. (4) Explicit escalation triggers — define conditions under which the agent stops and asks for human input rather than guessing.
Why do larger context windows not fix the error compounding problem? Context windows prevent forgetting, but error compounding is a cumulative accuracy problem, not a memory problem. At 95% per-step accuracy, 100 steps gives 0.59% success regardless of context window size. More context means the agent can see further back, but it cannot retroactively correct wrong premises it accepted as true several steps ago.
What is pass@k and why does it matter for deployment? Pass@1 asks: did the agent succeed at least once in a single run? This is how most benchmarks work. Pass@k asks: did the agent succeed on every one of k independent runs? This is closer to what deployment actually requires. Users don’t run your agent once — they run it many times, in slightly different contexts. An agent that succeeds 60% of the time at pass@1 might fail every third user interaction in practice.