Clinical EvalIndependent benchmark analysis / Final npj Digital Medicine article, 2025
MedS-Bench assembles medical language tasks that extend beyond selecting an examination answer. Our analysis uses the final journal version and keeps the evaluation collection separate from MedS-Ins, its associated training dataset. The central question is what each metric can actually support: exact or parsed label agreement, entity extraction, or similarity to reference wording. The task atlas exposes those distinctions before any model comparison. Published NER results are shown within one metric family; we do not combine unrelated measures into an invented overall ranking or treat a high closed-set score as evidence of a complete clinical workflow.
01 / What is being tested?
Data origin. A collection of existing biomedical and clinical datasets, including examinations, text records and literature-derived resources; provenance and access vary by constituent dataset. [1][2][3][4][5][6]
Final paper and official data card; README retains an earlier count of 39.
Figure 1; data-card Introduction [1][2][3]Final journal figure caption.
Figure 1 [1]Data-card taxonomy groups explanation and rationale together.
Introduction [2]MCQA is the explicitly separate zero-shot condition.
Results: evaluation settings [1]The associated training collection’s instance count is not MedS-Bench evaluation size.
Introduction [3]Identify source dataset, task definition and required output format.
[2]Use the authors’ selected cases where reproducing reported results.
[2][4]Preserve the study’s shot setting and instruction format.
[1]Retain output parsing and metric definitions with the result.
[1][5][6]02 / Measurement
Higher is better within a fixed metric and task.
Label tasks use accuracy, entity extraction uses F1, and generated text uses reference-overlap measures. Keep these families separate.
accuracy = correct/N; entity F1 = 2TP/(2TP + FP + FN)
Match the selected split, task, prompt examples and output parser. The same numeric value has different meanings across metric families. [1][6][5]
03 / Measured evidence
Paper-reported results / selected rows
Final 2025 journal paper, named-entity recognition tasks; few-shot evaluation.
Paper-reported historical results. These averages concern the NER task set only, not the full benchmark. No numeric uncertainty intervals are supplied.
Source: Table 6 [1]
04 / Our original analysis
The paper describes SEER treatment choices as eight categories and clinical outcome tasks as binary labels. [1]
Interpret the score as agreement within that output space; it does not evaluate a complete individualized plan.
The pinned information-extraction code tests whether normalized reference text occurs in the response. [5]
Additional material can leave this metric unchanged. Review contradictions separately when the intended application requires that distinction.
05 / Scope of the evidence
Final paper/data-card composition differs from the current README; this page fixes the final journal version. [1][3][2]
Reference overlap and label agreement do not independently establish medical safety or utility. [1]
Source resources have different origins and conditions; the collection should not be described as one uniform real-patient cohort. [1][2]
Pinned public scripts are observable implementations; this analysis does not claim every historical result used the identical commit. [5][6]
Evidence trail
Chaoyi Wu and colleagues · npj Digital Medicine. Final journal version: benchmark composition, evaluation settings, task-specific metrics and results.
MAGIC / Henrychur. Official task taxonomy and distinction between full source data and the reproduction split.
MAGIC: Medical Artificial General Intelligence Consortium. Author code; distinguishes training instances from benchmark data. README retains an earlier source-dataset count.
MAGIC / Henrychur. Published selected examples used for reproduction; do not silently substitute fresh random samples.
MAGIC. Current public scorer normalizes text and checks reference containment in the response.
MAGIC. Current public script aggregates entity TP, FP and FN within tasks and averages task F1 values.
Benchmark authors retain authorship of their work. This publication provides independent analysis; published rows are not Arcophos evaluation runs. Editorial method.