Detecting LLM Agent Contradictions Using NLI and Total Variance

This article presents a Python implementation to detect and diagnose logical contradictions in the outputs of large language model (LLM) agents using the Total Variance formula and natural language inference (NLI).

Details

Large language model (LLM) agents are known to be non-deterministic, meaning they can produce different outputs for the same query across multiple runs. While measuring the embedding similarity across runs can detect high variance, it fails to capture the critical distinction between inconsistent outputs and logically contradictory outputs. This article presents a Python implementation that combines the Total Variance formula from the arXiv paper 'Evaluating Stochasticity in Deep Research Agents' and natural language inference (NLI) to detect and diagnose logical contradictions in LLM agent outputs. The Total Variance metric quantifies the overall variance across multiple runs, while the NLI component specifically identifies cases where the outputs are logically opposite. This approach is particularly important for safety-critical applications of AI, such as in the medical, legal, or financial domains, where logical consistency is paramount.