# Production emergency: containing a $40K/day agent cost explosion

Late-stage platform, production multi-agent system.

Incident response · priced after triage.

The $40K/day figure is a projected run-rate, not a claim that a full day of that spend was invoiced. This write-up describes one production incident and the controls shipped during the response.

## Problem

A multi-agent pipeline in production began spending at a rate projected to exceed $40K per day — a $1.2M/month run-rate if left uncorrected — with no per-session budget controls and no circuit breaking at the tool-call layer.

## What shipped

- Led triage and spend attribution first: per-agent, per-model, and per-tool cost mapping before touching the live system.
- Shipped circuit breakers at the tool-call layer with per-agent and per-tool thresholds.
- Enforced per-session spend caps, keyed to task type, at runtime.
- Deployed retrospective governance: deterministic policy validation on every tool call, so spend is blocked before execution rather than reviewed after the bill lands.

## Reported outcomes

- Projected $40K+/day burn contained within 72 hours of engagement start.
- Zero recurrence since.
- The same controls shipped into the client's standard deployment path.

Anonymized client work, reported by Nick Cerutti. Client identities and raw operational records are not published.

## What to check in your system

A cost dashboard can explain yesterday's bill. Runtime limits address a different problem: stopping an agent or tool call before it exceeds an agreed budget. The incident work combined attribution with enforcement, then put those controls on the normal deployment path.

- Can you attribute spend to the agent, model and tool involved?
- What stops a session from retrying indefinitely?
- Are those limits part of the standard deployment, or an incident-only patch?

Stack: AWS Bedrock, AgentCore, policy-as-code tool-call validation

[Related engagement](https://nickcerutti.com/services/agent-production-audit) · [Free tool](https://nickcerutti.com/tools/agent-burn-calculator)

Canonical page: https://nickcerutti.com/work/emergency-40k