Benchmarking 3 Local LLMs on 50 Factual Questions
The author built an open-source hallucination benchmark for local large language models (LLMs) and tested 3 models - llama3.2, mistral, and phi3 - on 50 factual questions across 5 categories. The results show llama3.2 performed the best with 94% accuracy, while the other two models scored 88% and 86% respectively.
Why it matters
This benchmark provides a useful tool for evaluating the factual knowledge and reliability of local LLMs, which is crucial as these models are increasingly used in high-stakes applications.
Key Points
- 1Tested 3 local LLMs - llama3.2, mistral, phi3 - on 50 factual questions
- 2llama3.2 achieved 94% accuracy, outperforming mistral (86%) and phi3 (88%)
- 3Tested 4 prompting techniques but found no significant improvement in accuracy
- 4Shared the benchmark dataset and code for others to use
Details
The author built an open-source benchmark to test the reliability of local large language models (LLMs) in answering factual questions. They tested 3 models - llama3.2, mistral, and phi3 - on a dataset of 50 questions across 5 categories, running the tests fully locally using the Ollama tool. The results showed llama3.2 achieved the highest accuracy at 94%, correctly answering 47 out of 50 questions. The other two models scored 88% and 86% respectively. The author also tested 4 different prompting techniques, including chain-of-thought, self-consistency, and RAG grounding, but found they did not significantly improve the models' performance, indicating the bottleneck is the difficulty of the questions rather than the prompting strategy. The benchmark dataset and code have been shared publicly for others to use in testing the reliability of their own LLMs.
No comments yet
Be the first to comment