Optimizing Context Window Size for Better LLM Output

The article discusses how increasing the context window size in an AI pipeline did not improve the output quality. The author explains how irrelevant content can degrade model performance, and how context selection is more important than just increasing the window size.

Details

The article describes an AI pipeline that stitches context from multiple repositories, calls a large language model (LLM) with a chain-of-thought prompt, and posts the root cause analysis to Slack and Jira. The author noticed that the output quality dropped at some point, and upon investigation, found that the actual repository sizes were vastly different, with one legacy repository having 7.9M tokens compared to the frontend and backend repositories having 527k and 311k tokens, respectively. The fixed 50/35/15 budget split was loading the same proportion of irrelevant code regardless of the ticket type. The author explains that models don't attend uniformly across long contexts, and irrelevant content can degrade output quality. To address this, they implemented label-based routing to allocate the context budget based on the ticket type, and restructured the prompt to focus the model's attention on the relevant ticket information. The goal was to use deterministic pre-filtering before the LLM sees any code, so that the model sees less irrelevant content and produces better output.