Blitzy Outperforms GPT-5.4 on SWE-Bench Pro

This article compares the performance of Blitzy, an agentic software development platform, and GPT-5.4, the state-of-the-art language model, on the SWE-Bench Pro coding benchmark. Blitzy achieved a 66.5% score, outperforming GPT-5.4's 57.7% score.

Details

The article discusses the growing importance of agentic IDE tooling and 'vibe coding' in software development, but notes that enterprise systems are not easily disrupted by these trends. For complex enterprise codebases, the model alone is not enough - the agent harness or orchestration layer plays a crucial role. Blitzy, an agentic software development platform, recently achieved a 66.5% score on the SWE-Bench Pro Public benchmark, outperforming the current state-of-the-art model, GPT-5.4, which scored 57.7%. The article highlights that the SWE-Bench Pro benchmark is run by Scale AI, a company that primarily sells data to model owners and has no incentive to validate harnesses. However, recent tests have shown that a harness can offer significant improvements in performance over base models alone, even for advanced language models like Gemini 3.1 Pro, Claude Opus 4.6, and GPT 5.4. Blitzy's platform is designed specifically for enterprise software development, with a focus on collaborative analysis and detailed technical specifications, rather than targeting individual developers.