Production case study · Nick Cerutti
Production emergency: containing a $40K/day agent cost explosion
Projected $40K+/day burn contained within 72 hours. No recurrence reported during the engagement.
Late-stage platform, production multi-agent system. Incident response · priced after triage.
Context and scope
The $40K/day figure is a projected run-rate, not a claim that a full day of that spend was invoiced. This write-up describes one production incident and the controls shipped during the response.
A multi-agent pipeline in production began spending at a rate projected to exceed $40K per day — a $1.2M/month run-rate if left uncorrected — with no per-session budget controls and no circuit breaking at the tool-call layer.
What shipped
- Led triage and spend attribution first: per-agent, per-model, and per-tool cost mapping before touching the live system.
- Shipped circuit breakers at the tool-call layer with per-agent and per-tool thresholds.
- Enforced per-session spend caps, keyed to task type, at runtime.
- Deployed retrospective governance: deterministic policy validation on every tool call, so spend is blocked before execution rather than reviewed after the bill lands.
Outcomes
- Projected $40K+/day burn contained within 72 hours of engagement start.
- No recurrence reported during the engagement.
- The same controls shipped into the client's standard deployment path.
Sources and scope
author reported · Reviewed 2026-09-22
Client outcomes. Most client identities and raw operational records are not public; Softograph is named, and the remaining clients stay anonymous. $200K/year is aggregate savings across engagements. $40K/day was projected exposure; 72 hours describes that incident.
What to check in your system
A cost dashboard can explain yesterday's bill. Runtime limits address a different problem: stopping an agent or tool call before it exceeds an agreed budget. The incident work combined attribution with enforcement, then put those controls on the normal deployment path.
- Can you attribute spend to the agent, model and tool involved?
- What stops a session from retrying indefinitely?
- Are those limits part of the standard deployment, or an incident-only patch?
Stack: AWS Bedrock · AgentCore · policy-as-code tool-call validation
Working on a similar problem?
Two weeks: an independent read of cost, reliability, and governance across your agent stack, with every finding tied to a dollar figure.
Agent Production Audit · from $12K