Gemma4 vs Claude Code: I Tried the Switch. Here's What Broke First.

The article compares the performance of Gemma4, a new open-source AI model, against Claude Code, a commercial AI coding assistant. It highlights Gemma4's impressive benchmark scores but finds issues with its reliability in real-world coding tasks.

Details

The article explores the author's experience of testing Gemma4, a new open-source AI model, in their actual development workflow. While Gemma4 initially performed well on single-file edits and writing fresh functions, it struggled when asked to refactor a module across multiple files. The model exhibited classic context collapse, generating changes to files that didn't exist or calling functions it had just deleted. In contrast, the author found that the commercial AI assistant Claude Code was able to complete the same refactoring task in a single shot. The article delves into the underlying issues, noting that Gemma4's high tool-use success rate on benchmarks is primarily for the 31B Dense model, while the more commonly used 26B MoE variant scores significantly lower. This suggests that the tool-calling problem may be worse than the benchmarks suggest. The article also mentions undocumented performance features in Gemma4, such as multi-token prediction heads, that could improve its capabilities in the future. However, the author argues that the reliability and consistency of Claude Code in real-world coding tasks is hard to replace, even if Gemma4 may outperform it on certain benchmarks.