Microsoft's MAI-DxO diagnostic orchestrator tops physicians on sequential SDBench (Science): 83% vs 22–36% on NEJM cases, open-sources the benchmark

Why it matters

【Key points】

Microsoft AI on Oct 22 (18:01 UTC) and in Science the same day: diagnostic orchestrator MAI-DxO beats physicians on sequential benchmark SDBench and open-sources parts of the eval. Companion vision piece in BMJ Digital Health and AI.

1. Method: SDBench makes cases interactive step-by-step diagnosis—start with limited info, choose questions/tests, score accuracy, evidence, and estimated cost. Cases: 304 complex NEJM CPC plus MIMIC-IV-ED. MAI-DxO runs like a panel of virtual clinicians debating diagnoses and trading off cost vs value.

2. Results (Microsoft): 83% on NEJM vs 22% physicians without references / 36% with textbooks-internet (~3.7× / 2.3×); 80% vs 47% on ED cases (~1.7×). Cost-tilted config kept 81% on NEJM at under half comparator cost. Gains held with newer models including GPT-5.5. Prospective work with Mayo Clinic and Beth Israel Deaconess.

3. Open-source: SDBench for selected NEJM cases (Gatekeeper, Judge, cost code).

【Verifiable】

https://microsoft.ai/news/building-medical-superintelligence/

https://www.science.org/doi/10.1126/science.aeh7930

Verge / Techmeme

https://agipulse.org/e/766

【Our read】

1. USMLE-style one-shot quizzes inflated health-AI scores; sequential diagnosis plus cost is the right eval direction. "Medical superintelligence" overshoots—the paper is orchestration and a benchmark, not full care pathways.

2. Google's AMIE already has a Lancet prospective clinic study (e/766); OpenEvidence claims ~two-thirds of US physicians use its search. MAI-DxO leads on a Science-backed controlled benchmark but trails AMIE on real-world evidence; the BIDMC study aims to close that gap.

3. If orchestration gains survive model swaps, hospitals may buy "panel workflows" rather than bet on one model. Open SDBench forces one ruler—and invites fights over weak physician baselines and hard cases.

4. Caveat: physician tool access, US-centric cost models, and "4× accuracy" headlines travel badly; regulatory and liability questions remain open.