The $47,000 AI Agent That Went Unnoticed for 11 Days

A multi-agent AI system designed to help users research market data got stuck in an infinite loop, costing $47,000 before anyone noticed. The system's modular architecture lacked proper orchestration and monitoring, allowing the issue to go undetected for over a week.

Details

The article describes a multi-agent AI system designed to help users research market data. The system used four coordinating agents (Research, Analysis, Verification, and Summary) that communicated via agent-to-agent message passing. On paper, this modular architecture seemed sound, but in practice, it lacked critical features like an orchestrator, shared memory, and cost monitoring. Two of the agents got stuck in an infinite loop, where the Analysis Agent would expand its output in response to the Verification Agent's clarification requests, leading to an endless cycle of revisions. Neither agent was malfunctioning, but the lack of a termination condition allowed the loop to continue unabated. The cost escalation was gradual, going from $127 in the first week to $18,400 in the fourth week, before crossing $47,000 total. The team only discovered the issue when reviewing the cloud bill, as there were no real-time cost alerts or monitoring dashboards in place. The deeper problem was the 11 days of silence, where the system appeared to be functioning normally despite the underlying issue.