Single answer
A value, a classification or a conclusion, graded against the reference with a stated tolerance.
LabMedBench
A benchmark of large language models in laboratory medicine
LabMedBench measures how well language models handle the work of a clinical laboratory specialist: calculations, interpretation of results, quality control, preanalytical problems, method validation, reference intervals, interference and laboratory management.
October 2026: expert validation in progress. Results will be released with the article.101open-response questions
8domains of laboratory medicine
13commercial and open-weight models
3 + expertsLLM judges, adjudicated by specialists
Models
ModelDeveloperLabMed Omniscience Index
Claude Fable 5.1Anthropicin review
Claude Opus 5.5Anthropicin review
DeepSeek V4 ProDeepSeekin review
Gemini 3.8 FlashGooglein review
GLM-5.3Z.aiin review
GPT-5.6 LunaOpenAIin review
GPT-6 AstraOpenAIin review
GPT-6 LunaOpenAIin review
GPT-6.1 SolOpenAIin review
Grok 4.7xAIin review
HY3Tencentin review
HY4-previewTencentin review
Nemotron 3.5 Super-VLNVIDIAin review
Listed alphabetically. Scores will appear here when the article is published.
LabMed Omniscience Index
One number to judge a laboratory AI.
Accuracy tells you how often a model is right. LabOI also counts how often it is confidently wrong, the error that reaches a patient. If you look at one score, look at this one.
Right answer+1
"I don't know"0
Wrong answer−1
LabOI
100 × (right − wrong) ÷ questions
−100 guesses0trustworthy +100
Penalises hallucinationA fluent wrong answer costs a point.
Rewards restraintHolding back is never punished.
ComparableOne scale for every model, with a 95% confidence interval.
About
Many questions ask for a value or a conclusion that cannot validly be given for the case shown, for example because the sample, the calibration or the comparison data are unsuitable. A model gets these right only by saying why the result should not be used.
Question types
Single answerCannot be answered as askedRequired actions
Cannot be answered as asked
The requested number or verdict is invalid for the case. Correct only if the answer says so and does not rely on it.
Required actions
Cases, several built from anonymised patient panels with controlled artefacts. The answer must cover every required action and make no forbidden statement.
Domains
- Calculations21
- Result interpretation16
- Quality control14
- Preanalytical phase and specimen integrity14
- Method validation11
- Laboratory management9
- Reference intervals9
- Analytical interference7
Grading
13models, one run each
1,313answers
3blinded LLM judges from different families
Expertssettle every disagreement
LabOIwith 95% confidence interval
No judge grades its own answers. Each question and reference answer is also checked by at least one specialist who did not write it.
Test set
The questions are kept private so that they cannot end up in model training data. Specialists who validate them receive access by invitation.