The Routing Pattern: How Smart Teams Use Fast and Capable Models

This article discusses a three-tier approach used by teams building AI agents at scale, where a fast triage model handles initial requests, a capable execution model handles complex tasks, and a human review tier handles anything that falls through the cracks.

Details

The article explains that teams building AI agents often start with the most capable model, but the costs add up quickly. Switching to a faster, cheaper model can lead to missing edge cases. The solution is to build infrastructure that routes intelligently between the fast and capable models. The three-tier approach involves a lightweight fast triage model that handles initial requests, a heavyweight capable execution model for complex tasks, and a human review tier for anything that falls through the cracks. This approach can dramatically reduce inference costs, as 80% of requests may be handled by the fast model. Speed is also important, as users notice latency, so the routing logic itself needs to be very efficient. Progressive summarization can help the capable model do less work and respond faster. The routing mindset shifts model selection to a runtime decision, measures success by cost per successful outcome, and separates the architecture into triage, execution, and review.