When Your AI Elaborates, It Forgets to Count

An AI-powered educational video pipeline encountered a bug where the narration mentioned 5 test points, but the visual showed 7. This was due to a 'plan-to-script semantic drift' where the AI made a locally good decision to add more examples for better pedagogy, but didn't update the count in the narration.

Details

The article describes an AI-powered educational video pipeline that automatically plans lessons, writes scripts, generates visuals, and narrates. The team encountered a bug where the narration said 'let's look at five test points' while the visual showed seven dots on a number line. This was not a hallucination, but a result of the AI's decision-making process. In the script generation stage, the AI added two extra points for better pedagogical value, but did not update the count in the narration. This 'plan-to-script semantic drift' happened because the prompt boundary between planning and writing created a gap where the count was decided in one context and referenced in another. The team's initial instinct to build a verification gate was not robust enough to handle diverse visual types. Instead, they added a convergence condition to the script generation prompt, requiring the AI to ensure that every quantitative claim in the narration exactly matches the visual. This allowed the AI to make pedagogical improvements while maintaining accuracy, rather than prescribing a fixed process that would limit the AI's ability to teach better.