Benchmarking OpenAI, Anthropic, and Cohere for Bulk Content Generation

The author tested the performance of OpenAI, Anthropic, and Cohere APIs for a bulk content generation use case, evaluating factors like output quality, cost, latency, and error rate at scale.

Details

The author had a content pipeline that required processing 10,000 articles per month, and they evaluated three major LLM APIs - OpenAI, Anthropic, and Cohere - to determine the best fit for their use case. Rather than relying on existing benchmarks, which were often outdated or lacked real-world relevance, the author ran their own tests over 3 weeks, making 4,200 requests. The key evaluation criteria included output quality on structured content, cost per 1,000 words, latency (both p50 and p95), instruction adherence, and error rate over volume. The author did not test capabilities like coding tasks, reasoning, or multimodal inputs, as those were already well-covered in other benchmarks. The test setup involved a simple Node.js harness that logged results to a SQLite database, ensuring a consistent and measurable comparison across the three providers.