The Illusion of Readiness: Stress Testing Large Frontier Models on Multimodal Medical

The paper presents a critical evaluation of large frontier models (LFMs), such as GPT-5, on multimodal medical benchmarks, asserting that high leaderboard scores create an "illusion of readiness" in health AI. Through a series of targeted stress tests, researchers demonstrate that these models frequently rely on shortcut learning and brittle test-taking strategies — for instance, guessing correctly even when key visual input is removed, or producing medically sound but fabricated reasoning for wrong answers. The paper highlights that current benchmarks are fundamentally flawed, often rewarding superficial pattern matching over genuine medical understanding and robust reasoning, underscoring the urgent need for new evaluation standards that prioritise reliability and trustworthiness for real-world healthcare deployment.