Why AI Agents Fail in Production: The 90% Problem (18-min Podcast)

An 18-minute episode on why AI agents that work in a demo stop working in production, and why the fix is usually code around the model, not a better model.

The numbers behind it

• 40% → 60% pass rate on the same test cases with the same GPT-4-class model, from code changes alone. Seven of the eight fixes cost zero extra LLM calls (full case study).
• 1,421 lines: the size of the while(true) loop at the center of Claude Code’s leaked query.ts. Most of it is not the model call: context compression, error handling and recovery.
• 250,000 wasted API calls a day: what Claude Code’s source notes were being burned before a circuit breaker stopped after three consecutive failures.
• 0/2 → 3/3: Claude Code’s background memory extraction, before and after a verification step that checks the result and retries with the failure reason.

What the episode covers

• The five pillars of a production agent (3:08) and why context and memory are where most failures happen (4:06).
• Lost in the middle (5:04): irrelevant context dilutes attention. Sub-agents with isolated context keep the main loop clean (5:46).
• Tiered context compression (8:06): summarise old turns first, drop more aggressively only when needed.
• Cleaning errors before the model sees them (11:07): parse the stack trace in code and hand the model a short, actionable message.
• Open loops vs closed loops (13:09): a verification step that only logs a failure is an open loop. A closed loop feeds the failure back and retries.
• “Features exist” vs “loops are closed” (14:02): agents can ship SQLite memory and full-text search and still pass broken retrievals straight to generation.

Related

• Blog post: The AI Agent 90% Problem
• Case study: 40% to 60% with A/B data
• Short video: Why Your AI Agent Keeps Failing (7 min)
• Why Claude Code’s agent loop is 1,421 lines