Building an AI Lie Detector with Stylometric Forensics and Multi-Agent Debate
The author built an AI detector that combines stylometric analysis and a multi-agent LLM debate system to determine if text is human or AI-generated, with an accuracy of around 85%.
Why it matters
This approach to AI detection provides a more robust and explainable alternative to traditional black-box classifiers, which can be inaccurate and lack transparency.
Key Points
- 1Uses stylometric metrics like lexical diversity, sentence length, and punctuation density to extract forensic signals from the text
- 2Runs the text through three separate LLMs (Gemini, Claude, GPT-4) that debate and reach a consensus on whether it's human or AI-generated
- 3Provides a transparent, falsifiable process that shows the reasoning behind the classification, unlike black-box AI detectors
Details
The author wanted to build an AI detector that could explain why it flagged text as human or AI-generated, rather than just providing a confidence score. To achieve this, the workflow combines stylometric analysis (the forensic linguistics used to catch the Unabomber) and a multi-agent LLM debate system. First, it calculates various linguistic metrics like lexical diversity, sentence length variance, and punctuation density to extract forensic signals from the text. Then, it runs the text and metrics through three separate LLMs in sequence: Gemini makes the opening argument, Claude reviews the data and challenges or supports Gemini's argument, and GPT-4 acts as the judge, reviewing both arguments and issuing the final verdict. This multi-agent approach helps reduce hallucinations and provides a transparent, falsifiable process. The author tested the system on human writing, AI-generated text, and edge cases, achieving an accuracy of around 85%.
No comments yet
Be the first to comment