Gilles Penning

Last reviewed

Digital Health & AI in Medicine

What can AI do in medicine today?

The development of AI and its impact on medicine can be hard to follow. Here I bring together studies, benchmarks, milestones and related news, focusing on medical knowledge, clinical assistance and autonomous workflows. This should help you understand where we stand, how capabilities are changing, and which questions to ask when considering AI for healthcare delivery or new medical products.

The development of AI and its impact on medicine can be hard to follow. Here I bring together studies, benchmarks, milestones and related news, focusing on medical knowledge, clinical assistance and autonomous workflows. This should help you understand where we stand, how capabilities are changing, and which questions to ask when considering AI for healthcare delivery or new medical products.

Have a study, a question or a different perspective to share? I'd love to hear from you.

AI in Medicine 1950–2030

The 1950–2030 timeline shows 24 documented milestones. Positions on this chart are illustrative; the sources support the milestones, not real IQ scores.
General AI capabilityMedical knowledgeClinical assistanceAutonomy · METR context
2030?351001651950196019701980199020002010202020262030IQ for illustrationAutonomy: software tasks1 sec1 min1 hour1 day1950 · Turing: Imitation Game · Open source1956 · Dartmouth · Open source1966 · ELIZA · Open source1986 · Backpropagation · Open source1997 · Deep Blue · Open source2012 · AlexNet · Open source2016 · AlphaGo · Open source2020 · GPT-3 · Open source2022 · InstructGPT / RLHF · Open source2023 · GPT-4 report · Open source1975 · MYCIN · Open source2017 · Dermatology CNN · Open source2022 · MultiMedQA / Med-PaLM (Preprint) · Open source2023 · Med-PaLM (Nature) · Open source2021 · AlphaFold 2 · original paper · Open source2024 · AlphaFold 3 · Open source2025 · MedXpertQA v3 · Open source2026 · HealthBench Professional · Open source2024 · GPT-4: Diagnostic reasoning trial · Open source2025 · 2 studies: AMIE: Diagnostic dialogue, MedAgentBench · Open source2026 · 2 studies: MIRA, ResidencyRL · Open source2025 · METR: Software task horizon · Open source2026TODAY

Illustrative scale. Sources support milestones, not real IQ scores.

Data table
Documented milestones and studies on the 1950–2030 timeline
YearMilestoneTrackFindingSource
Turing: Imitation GameGeneral AI capabilityThe imitation game is proposed. A conceptual test of machine conversation, not an observed accuracy score.Mind 59(236), 433–460 ↗
DartmouthGeneral AI capabilityAI becomes a named research programme. The summer workshop is a historical event, not a performance result.Dartmouth ↗
ELIZAGeneral AI capabilityRule-based dialogue, without demonstrated understanding. ELIZA demonstrates conversational pattern matching rather than clinical reasoning.Communications of the ACM 9(1), 36–45 ↗
MYCINMedical knowledgeAntibiotic advice with rule-based explanations. MYCIN demonstrates a specific decision-support architecture; this source supplies no general clinical accuracy score.Computers and Biomedical Research ↗
BackpropagationGeneral AI capabilityHidden representations learned through backpropagation. The contribution is a training method, not a medical or intelligence score.Nature 323, 533–536 ↗
Deep BlueGeneral AI capability3.5–2.5 against Kasparov. Deep Blue wins a six-game match against the reigning world chess champion.IBM ↗
AlexNetGeneral AI capability18.9% top-5 error on LSVRC-2010. The source also reports 39.7% top-1 error; lower error is better.Advances in Neural Information Processing Systems 25 ↗
AlphaGoGeneral AI capability5–0 against Fan Hui. AlphaGo also wins 99.8% of games against other Go programs in the reported comparison.Nature 529, 484–489 ↗
Dermatology CNNMedical knowledgeComparable classification to 21 dermatologists. A neural network is compared with specialists on two binary skin-lesion classification tasks.Nature 542, 115–118 ↗
GPT-3General AI capability175 billion parameters; few-shot task adaptation. The model handles multiple language tasks using examples in the prompt rather than task-specific weight updates.arXiv 2005.14165v1; NeurIPS 2020 ↗
InstructGPT / RLHFGeneral AI capability1.3B InstructGPT preferred over 175B GPT-3. Human evaluators prefer the smaller instruction-tuned model on the tested prompt distribution.arXiv 2203.02155v1 ↗
MultiMedQA / Med-PaLM (Preprint)Medical knowledge67.6% MedQA accuracy for Flan-PaLM. This score belongs to Flan-PaLM, not the medically adapted Med-PaLM model.arXiv 2212.13138v1 ↗
GPT-4 reportGeneral AI capabilityAround the top 10% on a simulated bar exam. This is an academic exam comparison, not a clinical outcome or IQ score.arXiv 2303.08774v1 ↗
Med-PaLM (Nature)Medical knowledge67.6% MedQA accuracy for Flan-PaLM. This score belongs to Flan-PaLM, not the medically adapted Med-PaLM model.Nature 620, 172–180 ↗
GPT-4: Diagnostic reasoning trialClinical assistance76% with GPT-4 access vs 74% without. Adjusted difference +2 percentage points (95% CI −4 to 8; p=0.60): no significant assistance benefit.JAMA Network Open 7(10):e2440969 ↗
METR: Software task horizonAutonomy · METR contextAbout 50 minutes at 50% task success. Claude 3.7 Sonnet reaches this reported human-duration horizon on software tasks.arXiv 2503.14499v1 ↗
AMIE: Diagnostic dialogueClinical assistanceHigher ratings on 30/32 clinician and 25/26 actor dimensions. AMIE also has higher diagnostic accuracy in this simulated consultation comparison.Nature 642, 442–450 ↗
MIRAClinical assistance87.8% diagnosis accuracy vs 78.1% for physicians. The paired comparison uses 311 cases and four specialists; exact McNemar p=0.000287.Nature ↗
AlphaFold 2 · original paperMedical knowledge0.96 Å backbone error vs 2.8 Å. Median error is lower than the next-best method; lower is better.Jumper et al. · Nature ↗
AlphaFold 3Medical knowledgeProtein–ligand docking: better than Vina. On PoseBusters V1, AF3 predicted binding poses more accurately than Vina (p=2.27×10⁻¹³), without taking solved protein structures as input.Abramson et al. · Nature ↗
MedXpertQA v3Medical knowledge30.37% text accuracy for GPT-4o. This is the reported Text Avg result on MedXpertQA v3, not broad clinical accuracy.Zuo et al., Tsinghua University / Shanghai AI Laboratory ↗
HealthBench ProfessionalMedical knowledge59.0 rubric points for ChatGPT for Clinicians. The reported score is answer quality under this evaluation, not 59% correct diagnoses.OpenAI et al. ↗
MedAgentBenchClinical assistance69.67% task success for Claude 3.5 Sonnet v2. Success means completing the benchmark task on the first attempt under the authors’ agent harness.Jiang et al., Stanford ↗
ResidencyRLClinical assistance88% after training vs 81% for the base model. Seven percentage points more simulated conversations reach a diagnostic rubric score of at least 4/5.Liévin et al. ↗
MEDICINE MILESTONES2026
Medical Knowledge 20222023202420252026PubMedGPT (2.7B), 50.3%, MedQA. Source checked. Question Answering, results and hyperparameters tablesFlan-PaLM (540B), 67.6%, MedQA. Source checked. Abstract; best Flan-PaLM MedQA resultMed-PaLM 2, 86.5%, MedQA. Source checked. Table 4 and section 4.1 MedQAGPT-4 + Medprompt, 90.2%, MedQA. Source checked. Table 1, MedQA US 4-option; footnote **Med-Gemini (L 1.0), 91.1%, MedQA. Source checked. Abstract and section 4.1 MedQA resultsGPT-5.2, 94.2%, MedQA · 2026 study. Source checked. Fig. 2a; Methods: benchmark construction, generation parameters and MedQA scoringClaude Opus 4.6, 90.2%, MedQA · 2026 study. Source checked. Fig. 2a; Methods: benchmark construction, generation parameters and MedQA scoringGemini 3.1 Pro Preview, 97.4%, MedQA · 2026 study. Source checked. Fig. 2a; Methods: benchmark construction, generation parameters and MedQA scoringClinical Assistance 20222023202420252026GPT-3.5 Turbo, 15.54 points, HealthBench. Source checked. Table 7 mean columnGPT-4o (Aug 2024), 32.33 points, HealthBench. Source checked. Table 7 mean columnOpenAI o1, 42 points, HealthBench. Source checked. Table 7 mean columnGPT-4.1, 47.78 points, HealthBench. Source checked. Table 7 mean columnOpenAI o3, 59.9 points, HealthBench. Source checked. Table 7 mean columnClaude Opus 4.8: Different grader, not directly comparableClaude Opus 4.8: Different grader, not directly comparableGPT-5, 46.2 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3GPT-5.2, 45.9 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3Gemini 3.1 Pro, 43.8 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3Grok 4.20, 36.1 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3GPT-5.4 (Base), 48.1 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3Claude Opus 4.7, 47 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3ChatGPT for Clinicians (GPT-5.4), 59 points, Professional. Source checked. Figure 4 overall panel, printed bar labels; sections 4.1-4.3Claude Opus 4.7 (Anthropic grader), 51.9 points, Professional. Source checked. Grader: Claude Sonnet 4.6. pp. 228-229, section 8.14.1, Figure 8.14.A (grader in caption)Claude Opus 4.8, 56.9 points, Professional. Source checked. Grader: Claude Opus 4.8. Table 8.1.A, pp. 252-253 (score row p. 253); section 8.18.2, Figure 8.18.2.A, pp. 299-300Claude Mythos 5, 66 points, Professional. Source checked. Grader: Claude Opus 4.8. Table 8.1.A, pp. 252-253 (score row p. 253); section 8.18.2, Figure 8.18.2.A, pp. 299-300GPT-6 Astra (Anthropic run), 70.3 points, Professional. Source checked. Grader: Claude Opus 4.8. pp. 137-138, section 8.15, Figures 8.15.A/B and section 8.15.2GPT-6 Astra, 64.7 points, Professional. Source checked. Grader unconfirmed. Table 6, section 6.1; correction 2026-09-22GPT-6.1 Sol, 64.2 points, Professional. Source checked. Grader unconfirmed. Table 7, section 5.1Claude Sonnet 5.5, 69.2 points, Professional. Source checked. Grader: Claude Opus 4.8. pp. 137-138, section 8.15, Figures 8.15.A/B and section 8.15.2Autonomous Workflows 20222023202420252026Mistral 7B v0.3, 4%, MedAgentBench. Source checked. Table 2: Overall SRGemini 1.5 Pro, 62%, MedAgentBench. Source checked. Table 2: Overall SRGPT-4o, 64%, MedAgentBench. Source checked. Table 2: Overall SRGPT-4o mini, 56.33%, MedAgentBench. Source checked. Table 2: Overall SRClaude 3.5 Sonnet v2, 69.67%, MedAgentBench. Source checked. Table 2: Overall SRLlama 3.3 70B, 46.33%, MedAgentBench. Source checked. Table 2: Overall SRDeepSeek V3, 62.67%, MedAgentBench. Source checked. Table 2: Overall SRGemini 2.0 Flash, 38.33%, MedAgentBench. Source checked. Table 2: Overall SROpenAI o3-mini, 51.67%, MedAgentBench. Source checked. Table 2: Overall SRClaude Code / Opus 4.6, 19%, HealthAgentBench. Source checked. Leaderboard: Success RateCodex / GPT-5.3, 22%, HealthAgentBench. Source checked. Leaderboard: Success RateCodex / GPT-5.4, 28%, HealthAgentBench. Source checked. Leaderboard: Success RateClaude Code / Opus 4.7, 27%, HealthAgentBench. Source checked. Leaderboard: Success RateCodex / GPT-5.5, 42%, HealthAgentBench. Source checked. Leaderboard: Success RateClaude Code / Opus 4.8, 32%, HealthAgentBench. Source checked. Leaderboard: Success RateCodex / GPT-5.6-sol, 45%, HealthAgentBench. Source checked. Leaderboard: Success RateClaude Code / Opus 5, 55%, HealthAgentBench. Source checked. Leaderboard: Success RateCodex / GPT-6 Astra, 50.7%, HealthAgentBench v2. Source checked. Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria)Claude Code / Opus 5, 32%, HealthAgentBench v2. Source checked. Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria)Claude Code / Opus 5.5, 57.3%, HealthAgentBench v2. Source checked. Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria)Claude Code / Sonnet 5.5, 50.7%, HealthAgentBench v2. Source checked. Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria)

Medical Knowledge

97.4%

How well does AI answer medical exam questions?

Exam-style questions such as MedQA are close to saturation. On MedQA · 500 questions · Search enabled, the best confirmed result is Gemini 3.1 Pro Preview at 97.4% (model release 2026-02-19), with GPT-5.2 at 94.2% and Claude Opus 4.6 at 90.2%. These scores no longer separate the models widely. The open questions are in Clinical assistance and Autonomous workflows.

MedQA: medical exam questions · MedQA · 2026 study: 500 questions, search enabled · Accuracy (%)

Lines: best comparable result to dateDots: other models

Exam questions, not patient outcomes. Historical MedQA setups differ. Different tests; scores are not directly comparable.

Data table
Source-checked measurements for 1. Medical Knowledge
ModelBenchmarkScoreGraderProtocolRaw scoreTest settingStudy designReleaseSource
PubMedGPT (2.7B)MedQA50.3%See setupqualified—Exam / casesBenchmarkBioMedLM: a 2.7B parameter language model trained on biomedical text: Question Answering, results and hyperparameters tables ↗
Flan-PaLM (540B)MedQA67.6%See setupqualified—Exam / casesBenchmarkLarge Language Models Encode Clinical Knowledge: Abstract; best Flan-PaLM MedQA result ↗
Med-PaLM 2MedQA86.5%See setupqualified—Exam / casesBenchmarkTowards Expert-Level Medical Question Answering with Large Language Models: Table 4 and section 4.1 MedQA ↗
GPT-4 + MedpromptMedQA90.2%See setupqualified—Exam / casesBenchmarkCan Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine: Table 1, MedQA US 4-option; footnote ** ↗
Med-Gemini (L 1.0)MedQA91.1%See setupqualified—Exam / casesBenchmarkCapabilities of Gemini Models in Medicine: Abstract and section 4.1 MedQA results ↗
GPT-5.2MedQA · 2026 study94.2%See setupsame_protocol—Exam / casesBenchmarkGeneral-purpose large language models outperform specialized clinical AI tools on medical benchmarks: Fig. 2a; Methods: benchmark construction, generation parameters and MedQA scoring ↗
Claude Opus 4.6MedQA · 2026 study90.2%See setupsame_protocol—Exam / casesBenchmarkGeneral-purpose large language models outperform specialized clinical AI tools on medical benchmarks: Fig. 2a; Methods: benchmark construction, generation parameters and MedQA scoring ↗
Gemini 3.1 Pro PreviewMedQA · 2026 study97.4%See setupsame_protocol—Exam / casesBenchmarkGeneral-purpose large language models outperform specialized clinical AI tools on medical benchmarks: Fig. 2a; Methods: benchmark construction, generation parameters and MedQA scoring ↗

Sources

Clinical Assistance

48.1 points

How well does AI respond to clinical questions and conversations?

On HealthBench Professional, the best confirmed same-protocol result is GPT-5.4 (Base) at 48.1 points (model release 2026-03). On HealthBench Core, OpenAI o3 reaches 59.9 points.

HealthBench Core: clinical conversations · HealthBench Professional: professional clinical tasks · Rubric score (0–100 points)

HealthBench Professional: 0–100 points; no verified human baseline in this dataset.

Lines: best comparable result to dateDots: other models

Different grader, not directly comparable. Core and Professional are separate tests.

Data table
Source-checked measurements for 2. Clinical Assistance
ModelBenchmarkScoreGraderProtocolRaw scoreTest settingStudy designReleaseSource
GPT-3.5 TurboHealthBench15.54 pointsSee setupsame_protocol—Exam / casesBenchmarkHealthBench: Evaluating Large Language Models Towards Improved Human Health: Table 7 mean column ↗
GPT-4o (Aug 2024)HealthBench32.33 pointsSee setupsame_protocol—Exam / casesBenchmarkHealthBench: Evaluating Large Language Models Towards Improved Human Health: Table 7 mean column ↗
OpenAI o1HealthBench42 pointsSee setupsame_protocol—Exam / casesBenchmarkHealthBench: Evaluating Large Language Models Towards Improved Human Health: Table 7 mean column ↗
GPT-4.1HealthBench47.78 pointsSee setupsame_protocol—Exam / casesBenchmarkHealthBench: Evaluating Large Language Models Towards Improved Human Health: Table 7 mean column ↗
OpenAI o3HealthBench59.9 pointsSee setupsame_protocol—Exam / casesBenchmarkHealthBench: Evaluating Large Language Models Towards Improved Human Health: Table 7 mean column ↗
GPT-5Professional46.2 pointsGPT-5.4-lowsame_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
GPT-5.2Professional45.9 pointsGPT-5.4-lowsame_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
Gemini 3.1 ProProfessional43.8 pointsGPT-5.4-lowsame_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
Grok 4.20Professional36.1 pointsGPT-5.4-lowsame_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
GPT-5.4 (Base)Professional48.1 pointsGPT-5.4-lowsame_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
Claude Opus 4.7Professional47 pointsGPT-5.4-lowsame_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
ChatGPT for Clinicians (GPT-5.4)Professional59 pointsGPT-5.4-lowdifferent_protocol—Clinician-chat benchmarkBenchmarkHealthBench Professional: Evaluating Large Language Models on Real Clinician Chats: Figure 4 overall panel, printed bar labels; sections 4.1-4.3 ↗
Claude Opus 4.7 (Anthropic grader)Professional51.9 pointsClaude Sonnet 4.6different_protocol—Clinician-chat benchmarkBenchmarkClaude Opus 4.8 System Card: pp. 228-229, section 8.14.1, Figure 8.14.A (grader in caption) ↗
Claude Opus 4.8Professional56.9 pointsClaude Opus 4.8different_protocol—Clinician-chat benchmarkBenchmarkClaude Fable 5 & Claude Mythos 5 System Card: Table 8.1.A, pp. 252-253 (score row p. 253); section 8.18.2, Figure 8.18.2.A, pp. 299-300 ↗
Claude Mythos 5Professional66 pointsClaude Opus 4.8different_protocol—Clinician-chat benchmarkBenchmarkClaude Fable 5 & Claude Mythos 5 System Card: Table 8.1.A, pp. 252-253 (score row p. 253); section 8.18.2, Figure 8.18.2.A, pp. 299-300 ↗
GPT-6 Astra (Anthropic run)Professional70.3 pointsClaude Opus 4.8different_protocol74Clinician-chat benchmarkBenchmarkClaude Sonnet 5.5 System Card: pp. 137-138, section 8.15, Figures 8.15.A/B and section 8.15.2 ↗
GPT-6 AstraProfessional64.7 pointsNot explicitly statedqualified68.2Clinician-chat benchmarkBenchmarkGPT-6 Astra System Card: Table 6, section 6.1; correction 2026-09-22 ↗
GPT-6.1 SolProfessional64.2 pointsNot explicitly statedqualified67.2Clinician-chat benchmarkBenchmarkGPT-6.1 Sol System Card Addendum: Table 7, section 5.1 ↗
Claude Sonnet 5.5Professional69.2 pointsClaude Opus 4.8different_protocol77.1Clinician-chat benchmarkBenchmarkClaude Sonnet 5.5 System Card: pp. 137-138, section 8.15, Figures 8.15.A/B and section 8.15.2 ↗

Sources

Autonomous Workflows

57.3%

How reliably does AI complete multi-step tasks in simulated clinical systems?

HealthAgentBench v2: Claude Code / Opus 5.5 completed 43/75 tasks (57.3%). Different tasks and scoring from v1; no connected trend. v2 points use the paper date (25 October 2026), not model release.

MedAgentBench: simulated health-record tasks · HealthAgentBench: simulated clinical workflows · HealthAgentBench v2: 75 tasks · 9 categories · one attempt per task · Task success (%)

Lines: best comparable result to dateDots: other models

Simulated workflows, not autonomous care or patient outcomes. Different tests; scores are not directly comparable. No comparable trend established for this series.

Data table
Source-checked measurements for 3. Autonomous Workflows
ModelBenchmarkScoreGraderProtocolRaw scoreTest settingStudy designReleaseSource
Mistral 7B v0.3MedAgentBench4%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
Gemini 1.5 ProMedAgentBench62%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
GPT-4oMedAgentBench64%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
GPT-4o miniMedAgentBench56.33%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
Claude 3.5 Sonnet v2MedAgentBench69.67%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
Llama 3.3 70BMedAgentBench46.33%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
DeepSeek V3MedAgentBench62.67%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
Gemini 2.0 FlashMedAgentBench38.33%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
OpenAI o3-miniMedAgentBench51.67%See setupsame_protocol—Simulated workflowBenchmarkStanford ML Group - official benchmark: Table 2: Overall SR ↗
Claude Code / Opus 4.6HealthAgentBench19%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Codex / GPT-5.3HealthAgentBench22%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Codex / GPT-5.4HealthAgentBench28%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Claude Code / Opus 4.7HealthAgentBench27%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Codex / GPT-5.5HealthAgentBench42%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Claude Code / Opus 4.8HealthAgentBench32%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Codex / GPT-5.6-solHealthAgentBench45%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Claude Code / Opus 5HealthAgentBench55%See setupsame_protocol—Simulated workflowBenchmarkMicrosoft Research - official benchmark: Leaderboard: Success Rate ↗
Codex / GPT-6 AstraHealthAgentBench v250.7%See setupdifferent_protocol—Simulated workflowBenchmarkHealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗
Claude Code / Opus 5HealthAgentBench v232%See setupdifferent_protocol—Simulated workflowBenchmarkHealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗
Claude Code / Opus 5.5HealthAgentBench v257.3%See setupdifferent_protocol—Simulated workflowBenchmarkHealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗
Claude Code / Sonnet 5.5HealthAgentBench v250.7%See setupdifferent_protocol—Simulated workflowBenchmarkHealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗

Sources

Glossary

METR
METR: Software task horizon
MedQA
MedQA (USMLE 4-Option)
MedXpertQA
MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
AMIE
AMIE: Diagnostic dialogue
HealthBench
HealthBench: Evaluating Large Language Models Towards Improved Human Health
MIRA
MIRA: Diagnosis and actions in a virtual EHR
ResidencyRL
ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
MedAgentBench
MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents

Topic / subject

Capability
AI task and benchmark performance, not automatically clinical validity.
Clinical Evidence
Clinical evaluation, including early or simulated clinical studies; see test setting.
Real-World Use
Documented routine use in healthcare, independent of evidence for better outcomes.
Discovery
Biomedical discoveries and research enabled by AI.
Context
Historical or related context outside the four core medical AI themes.
Unclassified
Topic has not yet been classified.

Test setting - what was actually tested

Exam / cases
Questions, crafted cases or published teaching cases, without direct live patient care.
Clinician-chat benchmark
Offline, rubric-scored clinician–AI conversations or physician-authored clinical tasks (e.g. HealthBench Professional), often selected from real clinician chats. The tested model answers archived or curated tasks, not patients in a prospective clinical trial; the origin of a conversation does not prove treatment benefit.
Simulated workflow
Agent or diagnostic actions performed in a sandbox, including replayed EHR records.
Patient actors
Trained actors or standardized patients instead of patients receiving real care.
Retrospective records
Previously collected genuine patient information evaluated after the fact.
Prospective patients
Real patients enrolled and tested prospectively under a predefined study plan.
Routine care
Documented ongoing use in ordinary healthcare delivery; not an announced launch or one-off pilot.
Preclinical / animal
Non-human preclinical experiments including animals or phantoms.
Laboratory / research
Laboratory or computational-science benchmarks, not care of patients.
Non-clinical
Method, policy, historical or non-medical context without a patient evaluation.
Setting unknown
The actual testing environment is not sufficiently evidenced.

Study design - how the test was conducted

Benchmark
Standardized questions or tasks scored against a reference, potentially simulated.
Retrospective evaluation
Analysis of an existing dataset without prospective patient enrollment for the test.
Retrospective simulation
Historical cases replayed in a virtual environment; not prospective treatment.
Prospective single-arm
Prospective real-patient study with no concurrent control arm.
Prospective observational
Pre-planned observation of real patients without random treatment assignment.
Randomized simulation
Randomized vignette or actor experiment, not a clinical patient RCT.
Clinical RCT
Randomized controlled intervention study with real patients.
Routine observational
Observational data from a deployed tool in routine clinical care.
Case report / series
Description of one or several patient cases, not a standardized benchmark or controlled trial.
Historical event
Historical announcement, development or award rather than a study design.
Other design
Study or event not captured by the predefined design categories.
Design unknown
Design has not yet been established from methods.

Milestone marks editorial significance, not stronger clinical evidence. Peer review, model scores, real-patient testing and patient benefit are independent.

Feedback