Structured Stream Parsing for Instant AI Interactive Fiction

This article discusses how Novellum, an interactive fiction platform, overcame challenges in streaming LLM responses by implementing a reactive stream scanner that processes the output in a structured way.

Details

The article explains that when building the Novellum interactive fiction player, the initial approach of simply streaming the LLM output did not work well. The LLM response contains not just dialogue, but also narrative elements like character expressions, music moods, and scene descriptions, all of which need to be handled separately. The authors developed a reactive stream scanner that processes the output chunk-by-chunk, emitting partial dialogue tokens immediately and accumulating semantic tags until the closing tag is found. This allows for immediate downstream actions like streaming dialogue word-by-word, changing background music, updating character expressions, and queuing async scene image generation. The article also discusses edge cases like handling UTF-8 safety, ambiguous brackets, and coordinating speaker state updates. Implementing this structured stream parsing approach requires careful coordination across multiple components, including prompt engineering, server-side scanning, typed WebSocket protocols, and frontend projection systems.