Medical Knowledge
97.4%How well does AI answer medical exam questions?
MedQA: medical exam questions · MedQA · 2026 study: 500 questions, search enabled · Accuracy (%)
Lines: best comparable result to dateDots: other models
Exam questions, not patient outcomes. Historical MedQA setups differ. Different tests; scores are not directly comparable.
Data table
Clinical Assistance
48.1 pointsHow well does AI respond to clinical questions and conversations?
HealthBench Core: clinical conversations · HealthBench Professional: professional clinical tasks · Rubric score (0–100 points)
HealthBench Professional: 0–100 points; no verified human baseline in this dataset.
Lines: best comparable result to dateDots: other models
Different grader, not directly comparable. Core and Professional are separate tests.
Data table
Sources
Autonomous Workflows
57.3%How reliably does AI complete multi-step tasks in simulated clinical systems?
MedAgentBench: simulated health-record tasks · HealthAgentBench: simulated clinical workflows · HealthAgentBench v2: 75 tasks · 9 categories · one attempt per task · Task success (%)
Lines: best comparable result to dateDots: other models
Simulated workflows, not autonomous care or patient outcomes. Different tests; scores are not directly comparable. No comparable trend established for this series.
Data table
| Model | Benchmark | Score | Grader | Protocol | Raw score | Test setting | Study design | Release | Source |
|---|---|---|---|---|---|---|---|---|---|
| Mistral 7B v0.3 | MedAgentBench | 4% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| Gemini 1.5 Pro | MedAgentBench | 62% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| GPT-4o | MedAgentBench | 64% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| GPT-4o mini | MedAgentBench | 56.33% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| Claude 3.5 Sonnet v2 | MedAgentBench | 69.67% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| Llama 3.3 70B | MedAgentBench | 46.33% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| DeepSeek V3 | MedAgentBench | 62.67% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| Gemini 2.0 Flash | MedAgentBench | 38.33% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| OpenAI o3-mini | MedAgentBench | 51.67% | See setup | same_protocol | — | Simulated workflow | Benchmark | Stanford ML Group - official benchmark: Table 2: Overall SR ↗ | |
| Claude Code / Opus 4.6 | HealthAgentBench | 19% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Codex / GPT-5.3 | HealthAgentBench | 22% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Codex / GPT-5.4 | HealthAgentBench | 28% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Claude Code / Opus 4.7 | HealthAgentBench | 27% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Codex / GPT-5.5 | HealthAgentBench | 42% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Claude Code / Opus 4.8 | HealthAgentBench | 32% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Codex / GPT-5.6-sol | HealthAgentBench | 45% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Claude Code / Opus 5 | HealthAgentBench | 55% | See setup | same_protocol | — | Simulated workflow | Benchmark | Microsoft Research - official benchmark: Leaderboard: Success Rate ↗ | |
| Codex / GPT-6 Astra | HealthAgentBench v2 | 50.7% | See setup | different_protocol | — | Simulated workflow | Benchmark | HealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗ | |
| Claude Code / Opus 5 | HealthAgentBench v2 | 32% | See setup | different_protocol | — | Simulated workflow | Benchmark | HealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗ | |
| Claude Code / Opus 5.5 | HealthAgentBench v2 | 57.3% | See setup | different_protocol | — | Simulated workflow | Benchmark | HealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗ | |
| Claude Code / Sonnet 5.5 | HealthAgentBench v2 | 50.7% | See setup | different_protocol | — | Simulated workflow | Benchmark | HealthAgentBench v2: Section 4.1, PDF/printed p. 10 (exact numerators); Figure 4 p. 11 (rounded bars); Table 4 p. 23 (category scores); Tables 2-3 p. 21 (task counts/criteria) ↗ |
Glossary
- METR
- METR: Software task horizon
- MedQA
- MedQA (USMLE 4-Option)
- MedXpertQA
- MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
- AMIE
- AMIE: Diagnostic dialogue
- HealthBench
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- MIRA
- MIRA: Diagnosis and actions in a virtual EHR
- ResidencyRL
- ResidencyRL: Reinforcement Learning in Simulated Clinical Environments
- MedAgentBench
- MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents
Topic / subject
- Capability
- AI task and benchmark performance, not automatically clinical validity.
- Clinical Evidence
- Clinical evaluation, including early or simulated clinical studies; see test setting.
- Real-World Use
- Documented routine use in healthcare, independent of evidence for better outcomes.
- Discovery
- Biomedical discoveries and research enabled by AI.
- Context
- Historical or related context outside the four core medical AI themes.
- Unclassified
- Topic has not yet been classified.
Test setting - what was actually tested
- Exam / cases
- Questions, crafted cases or published teaching cases, without direct live patient care.
- Clinician-chat benchmark
- Offline, rubric-scored clinician–AI conversations or physician-authored clinical tasks (e.g. HealthBench Professional), often selected from real clinician chats. The tested model answers archived or curated tasks, not patients in a prospective clinical trial; the origin of a conversation does not prove treatment benefit.
- Simulated workflow
- Agent or diagnostic actions performed in a sandbox, including replayed EHR records.
- Patient actors
- Trained actors or standardized patients instead of patients receiving real care.
- Retrospective records
- Previously collected genuine patient information evaluated after the fact.
- Prospective patients
- Real patients enrolled and tested prospectively under a predefined study plan.
- Routine care
- Documented ongoing use in ordinary healthcare delivery; not an announced launch or one-off pilot.
- Preclinical / animal
- Non-human preclinical experiments including animals or phantoms.
- Laboratory / research
- Laboratory or computational-science benchmarks, not care of patients.
- Non-clinical
- Method, policy, historical or non-medical context without a patient evaluation.
- Setting unknown
- The actual testing environment is not sufficiently evidenced.
Study design - how the test was conducted
- Benchmark
- Standardized questions or tasks scored against a reference, potentially simulated.
- Retrospective evaluation
- Analysis of an existing dataset without prospective patient enrollment for the test.
- Retrospective simulation
- Historical cases replayed in a virtual environment; not prospective treatment.
- Prospective single-arm
- Prospective real-patient study with no concurrent control arm.
- Prospective observational
- Pre-planned observation of real patients without random treatment assignment.
- Randomized simulation
- Randomized vignette or actor experiment, not a clinical patient RCT.
- Clinical RCT
- Randomized controlled intervention study with real patients.
- Routine observational
- Observational data from a deployed tool in routine clinical care.
- Case report / series
- Description of one or several patient cases, not a standardized benchmark or controlled trial.
- Historical event
- Historical announcement, development or award rather than a study design.
- Other design
- Study or event not captured by the predefined design categories.
- Design unknown
- Design has not yet been established from methods.
Milestone marks editorial significance, not stronger clinical evidence. Peer review, model scores, real-patient testing and patient benefit are independent.