The Prompt Engineering Journey: Successes and Failures

The article discusses the author's experience with optimizing an AI agent to determine a product's manufacturing country from its barcode. It covers various prompt engineering approaches, their impact on accuracy and false confidence, and key lessons learned.

Details

The author built an AI-powered app called Mio that can determine a product's manufacturing country from its barcode. Over three weeks, they ran 108 benchmarks, tested 7 models, and iterated through 6 major prompt versions to optimize the agent's performance. The article highlights several key lessons. Firstly, the author learned that optimization is a multi-dimensional challenge, where a change that fails in one context can succeed in another. For example, anti-false-confidence rules that worked well on the Gemini 3 Flash model failed on the simpler Gemini 3.1 Flash Lite model. The author also cautions against adding fallback strategies that allow the model to easily provide low-quality answers, as this can lead to catastrophic failures. The brand-level fallback search, which the author thought would be a good idea, resulted in the worst performance in the project's history, with 13 false confidence cases. Ultimately, the author found that switching to a more advanced Gemini 3 Flash model and recalibrating the prompt led to a significant 20% accuracy improvement over the original production model. This underscores the importance of selecting the right model and continuously refining the prompt engineering approach.