Stop Paying for Slop: A Deterministic Middleware for LLM Token Optimization

This article introduces a Prompt Token Rewriter middleware that can compress prompts sent to large language models (LLMs) by 50-80%, reducing costs and inference time while maintaining deterministic behavior.

Details

As context windows for LLMs continue to grow, the token budgets are tightening. This article presents a solution called the Prompt Token Rewriter, a deterministic middleware that can significantly compress prompts before they are sent to the LLM. By removing conversational filler, redundant whitespace, and low-entropy 'slop', the middleware can reduce prompt size by 50-80%. This leads to lower costs, as users only pay for the 'signal' rather than the 'noise', and faster inference, as there is less data for the LLM to process. Importantly, the deterministic nature of the rewriter ensures stable and repeatable agent behavior, unlike approaches that rely on additional LLM calls. The middleware offers three preset levels of compression, allowing users to balance optimization and safety depending on their use case. This work is part of a broader effort to build a community-driven 'App Store' for agentic capabilities, decoupling logic from intelligence.