Analyzing 10,000 Voice AI Calls: LLM Rarely the Problem

The article discusses the findings from analyzing 10,000 voice AI calls, where the majority of failures were not due to the language model (LLM) but rather issues with the phone call medium, such as speech recognition, call flow, and infrastructure.

Details

The article presents the findings from analyzing 10,000 voice AI calls across customer support, appointment booking, and lead qualification use cases. Contrary to the initial assumption that most failures would come from the language model (LLM), the data showed that the majority of issues were related to the phone call medium itself. The top problems included speech recognition (STT) errors due to low-quality audio and accent variations, chaotic first 8 seconds of the call, interruption handling, extended silences, and tool call latency. Only about 15% of the failures were attributed to LLM-specific issues like hallucinations and instruction drift. The article discusses strategies to mitigate these challenges, such as customizing the STT vocabulary, setting expectations for transcription errors in the LLM prompt, and optimizing the call greeting to reduce latency and barge-in. Overall, the key takeaway is that voice AI systems need to go beyond just improving the language model and also address the fundamental limitations of the phone call medium.