AI Coding Benchmark: GPT-5.2 Codex vs Opus 4.5, Gemini & DeepSeek (Bug Fix, Refactor, Migration)

How do GPT-5.2, Codex, Claude Opus 4.5, DeepSeek V3.2, and Gemini 3 Pro handle real bug fixes, refactors, and migrations? I built a benchmark to find out, tracking correctness, cost, latency, and contract compliance across all five models.

In this first baseline run, each model was tested with identical prompts and constraints, exposing clear differences in reliability, recovery, and efficiency.

⏱️ TIMESTAMPS

00:00 Introduction – Why real engineering benchmarks matter
01:11 Baseline context & benchmark philosophy
01:57 Rules & constraints – execution, repair turns & output contract
03:24 Task overview – Bug Fix, Refactor, Migration
04:13 Benchmark prompt walkthrough (Bug Fix example)
04:53 Models tested & base token pricing context
06:07 Bug Fix results – correctness vs contract compliance
07:29 Refactor results – recovery & repair-turn behavior
09:02 Migration results – coordination & efficiency
10:09 Key takeaways – reliability, cost & automation readiness
11:37 Tracking results & future benchmark updates

🧪 MODELS TESTED

• GPT-5.2
• GPT-5.2 Codex (default reasoning)
• Claude Opus 4.5
• Gemini 3 Pro
• DeepSeek V3.2

All runs use temperature = 0 to ensure deterministic behavior.

🔍 WHAT THIS BENCHMARK SHOWS

◆ How models behave under strict automation rules
◆ Why format compliance matters as much as correct code
◆ How models differ in recovery ability after failure
◆ The trade-offs between cost, latency, and reliability
◆ Why baseline benchmarks matter before multi-turn workflows

⚠️ IMPORTANT CONTEXT

• Each task is a single controlled run per model
• Models are allowed one repair turn if the initial attempt fails
• Repair turns re-send the full prompt plus test-failure output
• This benchmark surfaces failure modes, not best-case performance

Future benchmarks will explore:

• Multi-run variance
• Multi-turn and agent workflows
• Higher reasoning modes (e.g. Codex x-high)

💬 WHAT SHOULD I TEST NEXT?

If you’d like to see:

• Other models added to this benchmark
• Multi-turn or agent-style benchmarks
• Different constraints (TDD, acceptance tests, iteration loops)

Drop your suggestions in the comments.

🌐 RESULTS & UPDATES

I’ll be publishing and updating benchmark results on my website:

👉 https://snapperai.io

Sign up for the newsletter to get updates on new benchmarks, models, and tooling.

▶️ WATCH NEXT

→ How the Creator of Claude Code Sets Up His Workflow:    • How the Creator of Claude Code Sets Up His...  
→ Claude Code Advanced Workflow Tutorial (Slash Commands & Subagents):    • How the Creator of Claude Code Uses Slash ...  
→ GPT-5.2 Codex vs Opus 4.5 — Tetris Build Test:    • GPT-5.2-Codex vs Opus 4.5: Tetris Build Te...  
→ GLM 4.7 vs Opus 4.5 vs GPT-5.2: One-Shot Build Test:    • GLM-4.7 vs Opus 4.5 vs GPT-5.2: One-Shot B...  

🔔 SUBSCRIBE

Subscribe for real-world AI coding benchmarks, workflow breakdowns, and hands-on tooling reviews.

🌐 Newsletter & templates: https://snapperai.io
🐦 X / Twitter: https://x.com/SnapperAI
🧑‍💻 GitHub: https://github.com/snapper-ai