Lessons Learned from Running a Multi-Agent AI System

The author shares their experience of transitioning from a conceptual multi-agent AI system to actually running it in production, highlighting the challenges and lessons learned along the way.

Details

The author initially designed a multi-agent AI system to separate responsibilities, use different models for different phases, and maintain human approval before implementation. However, when they started actually running the system, they encountered various practical challenges that were not apparent in the conceptual design. The biggest change was shifting from thinking in terms of a pipeline to an explicit runtime for the multi-agent system. This made the author realize that separating roles is not enough - you also need a robust runtime to make that separation viable. Real-world execution revealed gaps, such as the agents being unable to write files autonomously due to permission issues, and the orchestrator not providing enough context and decision-making capabilities for interactive sessions. The author also learned that not every task requires the full multi-agent pipeline, and that a task classification system can help optimize the process. Additionally, the author discovered that AI providers are not interchangeable and have different behaviors that need to be accounted for, which was not as apparent in the conceptual stage.