Building an AI Humanizer: why we stopped trying to fix prompts

The article discusses the limitations of using prompts to improve the naturalness of language model output, and how the authors instead focused on sentence-level rewriting to address issues like low sentence length variance and repeated syntactic patterns.

Details

The article discusses the authors' experience in building an 'AI Humanizer' to address the unnatural quality of language model output. They found that issues like low sentence length variance, shallow clause depth, high frequency of discourse markers, and predictable sentence openers were stronger signals of unnaturalness than vocabulary choice or tone. Prompt-based approaches failed to reliably fix these structural issues, as prompts primarily affected higher-level aspects like word choice and politeness, but not the local sentence-level patterns. The authors then reframed the problem, treating LLM output as raw material to be post-processed rather than as final text. They developed a two-stage pipeline: first optimizing for clarity and correctness in generation, then applying sentence-level rewriting techniques to adjust the distribution and flow of the text without changing the semantics. This approach allows them to measure and debug the changes, unlike the opaqueness of prompt tuning. The AI Humanizer within Dechecker implements this sentence-level rewriting approach as a controllable post-processing layer. The authors argue that this problem exists even without the need for AI detection, as uniform sentence structure can be tiring for human readers, and restoring natural irregularity is important for improving readability.