Stop Writing Unit Tests for Your AI Code. Write These 4 Evals Instead.
The article discusses why unit tests are not suitable for testing AI/LLM code and proposes 4 types of evaluations to ensure correctness instead.
Why it matters
This article provides a practical framework for testing and validating AI/LLM applications, which is crucial as these models become more widely adopted.
Key Points
- 1Unit tests are designed for deterministic functions, but LLMs are non-deterministic
- 2The right layer for AI correctness is evals, not unit tests
- 3Schema-validation evals, Canary evals, Regression evals, and Human-in-the-loop evals are recommended
- 4Evals should be used to test the semantic output of the model, not just the input/output contract
Details
The article argues that unit tests are not an appropriate way to test AI/LLM code because LLMs are inherently non-deterministic. Things like temperature, provider drift, model updates, and stochastic sampling can all cause the same input to produce different outputs. Instead, the author proposes 4 types of evaluations to ensure correctness: 1) Schema-validation evals to check the shape of the model's JSON output, 2) Canary evals to test the model's behavior on a set of known inputs, 3) Regression evals to detect regressions in model performance over time, and 4) Human-in-the-loop evals to get subjective feedback on the model's outputs. The key is to test the semantic correctness of the model's outputs, not just the input/output contract.
No comments yet
Be the first to comment