Build an Evaluation Harness for 184 AI Agent Prompts with Promptfoo

The article describes how to build an evaluation harness for a collection of 184 AI agent prompts using Promptfoo. The harness automatically scores the agents on criteria like task completion, workflow, character, output quality, and safety.

Details

The article discusses the need for an evaluation system to assess the quality and performance of the 184 AI agent prompts in the Agency-agents open-source collection. The author explains that while the prompts may look good on paper, there is no way to automatically verify if they actually produce useful and unbiased outputs. The evaluation harness built using Promptfoo addresses this by orchestrating three steps: loading the agent prompt as the system prompt, sending a task from a per-category YAML file, and scoring the output using a separate LLM acting as a judge. The judge scores the agent's output on five criteria: task completion, workflow adherence, character consistency, output quality, and safety/bias. The harness also extracts success metrics, critical rules, and deliverable templates from the agent markdown files to inform the scoring rubric. This allows the harness to provide a comprehensive and automated evaluation of the agent prompts, enabling regression testing and continuous improvement of the collection.