Implementing a Robust Watchdog for Long-Running AI Agents

The article discusses the author's experience with an OOM (Out of Memory) crash in a multi-agent AI system, and the steps taken to implement a reliable watchdog mechanism to prevent such issues in the future.

Details

The author runs a multi-agent AI system called the Pantheon, where 5 specialized AI agents (gods) are orchestrated by a central planner (Atlas). One of the gods experienced an OOM condition, which caused the process to silently die without any alert or log entry. This led to half-finished work, corrupted state files, and a system that appeared healthy from the outside but had a dead worker at its core. To address this issue, the author implemented a 3-layer watchdog system. The first layer uses launchd on macOS to automatically restart the agent process in case of a crash, handling clean restarts and exponential backoff to prevent restart storms. The second layer involves the agents writing heartbeat files every 15 seconds, which the orchestrator checks to detect stale agents. If a heartbeat is more than 60 seconds old, the agent is considered dead and can be restarted. The third layer is a central monitoring process that oversees the overall health of the system and can take appropriate actions, such as restarting agents or notifying the operator.