What Counts as Ground Truth When Machines Read About the Brain

Question: How have automated and semi-automated text-mining pipelines been validated against manual curation when extracting brain region–function associations from the neuroscience literature?

Answer

Taken together, the retrieved papers show that validation in this task is organized around recovering a curator's list, not around establishing that an extracted region–function association is correct. Three regimes dominate: one-sided recall against a systematic review conducted by the authors themselves, where text mining reached 98% recall at 36% precision and found three times as many targets as two person-weeks of curation [1]; graph-level precision/recall against a pre-existing curated knowledge graph, reported as relative gains rather than as agreement with an independently annotated sample [8]; and downstream utility or expert plausibility, where a pipeline is judged by whether its output improves question answering, graph topology, or expert ratings [9]. The comparator in every case is a manual resource whose own annotation conventions are unstated, whose granularity differs from study to study (over 800 anatomical terms in one platform [2], 106 neocortical fields in another [13]), and whose inter-annotator reliability is rarely reported. Because evaluation criteria demonstrably move reported performance by double digits [6], the field's published accuracy figures are not comparable across papers, and no retrieved study establishes construct validity for the associations themselves.

Recall against a curator's list dominates, leaving precision as a triage cost

The most direct head-to-head comparison of text mining against manual curation in this literature reports efficiency asymmetrically: 98% recall at 36% precision, framed explicitly as "only one target missed" rather than as a two-thirds false-positive rate [1]. That framing recurs in adjacent evidence-synthesis pipelines, where success is defined as retaining recall above a pre-set threshold while cutting workload — a 39.5% screening workload reduction at recall above 0.9 [22], 100% sensitivity with 90.8% specificity against a benchmarking set of published reviews [21], and 81% sensitivity in identifying eligible studies while additionally retrieving articles missed by manual screening [24]. Recall-first validation is defensible for screening, but it leaves the precision of the extracted association unmeasured, which is the quantity a knowledge-base consumer actually depends on.

Precision and recall against inherited knowledge graphs measure agreement with earlier curation

Where balanced metrics are reported, the reference standard is a prior curated graph rather than a fresh annotation: precision 0.80 and recall 0.81, presented as relative improvements of +6.67% and +15.71% over the existing resource [8], and precision 85.20% with recall 87.30% alongside a separate "expert validation" figure of 89.50% [11]. A more careful case reports an F1 of 0.84 only for entity extraction from unlabeled text, not for the relations that constitute the region–function association, while claiming comparability to a supervised method [9]. These numbers certify consistency with what earlier curators wrote, so any systematic convention in that earlier curation — which relation types count, how granular a region label must be — is inherited as ground truth rather than tested.

The manual reference standard is convention-bound and only moderately reproducible

Reported performance depends strongly on evaluation design: switching from overlap to strict span-boundary criteria lowered F1 from 0.60 to 0.48–0.52, and moving from a restricted candidate set to a realistic concept space lowered accuracy@1 from 0.90–0.94 to 0.56–0.69, with most false positives turning out to be valid concepts rather than hallucinations [6]. Agreement among models is likewise a salience signal, not corpus-convention correctness, and thresholds set on one entity type require target-domain revalidation under entity-type shift [3]. Human agreement sets an approximate ceiling that models do not reach on annotation tasks of comparable ambiguity, even for features humans agreed on reliably [7]. Disagreements between an LLM and a "gold standard" of human annotations were often not errors at all once reviewed [4]. Anatomical labelling itself shows the same ceiling: nine raters labelling the same brainstem structure reached only moderate agreement (mean Dice 0.7) [16]. The tools for reporting this properly exist — Krippendorff's alpha accommodates any number of raters and missing data in neuroimaging [18] — but no retrieved region–function pipeline reports it for its own gold standard.

Validity is increasingly argued from downstream utility and expert plausibility

Rather than measuring extraction error, several pipelines claim validity from what the extracted knowledge enables: answers to over 52% of neuroscience questions in PubMedQA and entity-augmented retrieval [9], multi-label classification at F1 0.856 built on multi-LLM-extracted records whose reliability is asserted via inter-model kappa of 0.822 [14], and outputs validated against a thirty-six-concept expert glossary scored by a council of ten local models [10]. One platform projects associations from 36 million articles onto a parcellated brain with no reported accuracy metric at all, presenting hypothesis generation as the use case [2]. Expert acceptability can also stand in for error rates, as when 14 domain experts each rated retained candidates more credible than rejected comparators [15]. An external criterion is available — meta-analytic concept maps from Neurosynth used as a validation target [17] and as the substrate for lesion-based decoding checked against NIHSS subscores [20] — but it is a literature-derived reference, so it inherits the same conventions it would be used to test.

Semi-automated pipelines are best validated as review-queue producers

The strongest validation designs retrieved treat automation as ranking rather than decision-making: a supervised scorer raises AUROC from 0.753 to 0.910 over raw multi-model agreement and, at a 0.95 precision target, selects 1,340 candidates at empirical precision 0.939 with candidate-level recall 0.592 [3]; targeted review of the highest-risk 15.6% of extraction cells raised accuracy from 91.70% to 95.22% [23]; and in a closely analogous meta-analytic selection task, manual image selection gave the most precise maps while heuristic selection served as a robust alternative for heterogeneous domains [19]. An LLM-assisted extraction pipeline validated against a human-adjudicated gold standard reached Krippendorff's alpha 0.606 versus 0.585 for human–human agreement, demonstrating that the validator itself can be calibrated [5]. This is the only regime in which "validated against manual curation" is a well-posed claim, because it specifies what the human does with the machine's output.

Conflicts and gaps

The papers disagree on whether human parity has been reached: one reports GPT-4o performing comparably with specially trained annotators at 0.91–0.97 on zero-shot prompts [4], while another finds models falling short of human reliability even where humans agreed reliably [7]. This cannot be adjudicated because the tasks, annotation conventions, and whether disagreements count as errors all differ. The apparent contradiction between "text mining is high-recall, low-precision" [1] and "text mining is balanced at precision ≈ recall ≈ 0.8" [8] is likewise unresolvable: the two use different reference standards, different relation definitions, and different units of comparison. No retrieved paper reports a purpose-built, independently double-annotated gold standard of region–function pairs with a reported agreement statistic, stratified by relation type and anatomical granularity, and several validate only entities rather than associations [9]. Authoritative resources in this space remain fully manual, with inclusion requiring convergent functional evidence and boundary-defining contrasts across 1,465 studies [13] and rigorous expert distillation for connectivity [12] — so the human comparator is a judgment-laden scholarly product, not an error-free label set.

What would change the answer

A shared benchmark of several hundred annotated paragraphs, with region–function pairs labelled by at least three neuroscientist curators under an explicitly written convention (granularity rules, boundary rules, permitted relation types) and adjudicated, reporting Krippendorff's alpha per relation type [18], would convert the current relative metrics into comparable ones. Reporting should be mandatory under both overlap and strict-boundary criteria and under both restricted and realistic label spaces, since these alone move F1 by more than ten points [6], and thresholds should be revalidated on a held-out subfield to test entity-type shift [3]. Construct validity would be settled by asking whether a pipeline's extracted associations predict an independent outcome — held-out meta-analytic maps [17] or behavioural deficit scores from lesion data [20] — rather than whether they match a prior list. Finally, a prospective randomised comparison of pipeline-assisted versus unaided curation of the same corpus, scoring both curator hours and adjudicated error rates on a sampled subset, would test the claim that motivates all of this work.

References

[1] Xavier Vasques, Renaud Richardet, Sean L Hill, David Slater, Jean-Cedric Chappelier, Etienne Pralong, Jocelyne Bloch, Bogdan Draganski, Laura Cif. "Automatic target validation based on neuroscientific literature mining for tractography". arxiv. 2025-03-05T00:00:00.000Z. URL: https://doi.org/10.3389/fnana.2015.00066

[2] Stefanovski, L., Bülau, K., Martin, L., Huettl, C., Langford, C., Palmer, J., Sacks, M., Deger, L., Pille, M., Schirner, M., Meier, J., Neudorfer, C., Horn, A., Solodkin, A., Thirion, B., Hofmann-Apitius, M., Jacobs, M., Tom Kodamullil, A., for the Alzheimer's Disease Neuroimaging Initiative,, Ritter, P.. "An Interactive Brain Atlas of Knowledge". bioRxiv. 2026-01-26T00:00:00.000Z. URL: https://doi.org/10.64898/2025.12.12.694043

[3] Shuheng Cao, Ruiqi Chen, Renjie Cao, Zhenhao Zhang, Siyu Zhang, Tingting Dan. "Beyond Agreement: Scoring Panel-Surfaced Biomedical Entity Candidates for Curator Triage". arxiv. 2026-06-01T00:00:00.000Z. URL: https://arxiv.org/abs/2605.30826

[4] Turner, M. D., Appaji, A., Ar Rakib, N., Golnari, P., Rajasekar, A. K., Rathnam K V, A., Sahoo, S. S., Wang, Y., Wang, L., Turner, J. A.. "Large Language Models Can Extract Metadata for Annotation of Human Neuroimaging Publications". bioRxiv. 2025-05-14T00:00:00.000Z. URL: https://doi.org/10.1101/2025.05.13.653828

[5] Maria Kunilovskaya, Gagan Bhatia, Lisa Sophie Albertelli, Yanran Chen, Christian Greisinger, Lotta Kiefer, Christoph Leiter, Subhadeep Roy, Tewodros Achamaleh, Muhammad Arslan Manzoor, Sebastian Pohl, Yufang Hou, Steffen Eger. "Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025". arxiv. 2026-09-03T00:00:00.000Z. URL: https://arxiv.org/abs/2606.02255

[6] Mazzucato, S., Seinen, T. M., Moccia, S., Micera, S., Bandini, A., van Mulligen, E. M.. "Advancements in Multilingual Biomedical Natural Language Processing: exploring Large Language Models for Named Entity Recognition and Linking". medRxiv. 2026-07-16T00:00:00.000Z. URL: https://doi.org/10.64898/2026.01.22.26344605

[7] Leonardo S. Goodall, Dor Shilton, Daniel A. Mullins, Harvey Whitehouse. "Large language models struggle with ethnographic text annotation". arxiv. 2026-01-21T00:00:00.000Z. URL: https://arxiv.org/abs/2601.12099

[8] Ali Sarabadani, Kheirolah Rahsepar Fard, Hamid Dalvand. "ExKG-LLM: Leveraging Large Language Models for Automated Expansion of Cognitive Neuroscience Knowledge Graphs". arxiv. 2025-07-09T00:00:00.000Z. URL: https://arxiv.org/abs/2503.06479

[9] Pralaypati Ta, Sriram Venkatesaperumal, Keerthi Ram, Mohanasankar Sivaprakasam. "Entity-Augmented Neuroscience Knowledge Retrieval Using Ontology and Semantic Understanding Capability of LLM". arxiv. 2025-10-28T00:00:00.000Z. URL: https://arxiv.org/abs/2506.03145

[10] Hamed Nejat, Alexander Maier, Jesse Spencer-Smith, André M. Bastos. "Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature". arxiv. 2026-06-05T00:00:00.000Z. URL: https://arxiv.org/abs/2606.05206

[11] Ali Sarabadani, Kheirolah Rahsepar Fard. "MultiCNKG: Integrating Cognitive Neuroscience, Gene, and Disease Knowledge Graphs Using Large Language Models". arxiv. 2026-06-23T00:00:00.000Z. URL: https://doi.org/10.22034/TMI.2026.244943

[12] Imam, F. T., Gillespie, T. H., Ziogas, I., Surles-Zeigler, M. C., Tappan, S., Ozyurt, B., Boline, J., de Bono, B., Grethe, J. S., Martone, M. E.. "Developing a Multiscale Neural Connectivity Knowledgebase of the Autonomic Nervous System". bioRxiv. 2024-11-28T00:00:00.000Z. URL: https://doi.org/10.1101/2024.10.25.620360

[13] Poliva, O.. "The Brain Encyclopedia Atlas Project (BEAP): A Literature-Synthesis-Derived Functional Atlas of the Human Brain". bioRxiv. 2026-05-26T00:00:00.000Z. URL: https://doi.org/10.64898/2026.05.05.722843

[14] Saja Al-Dabet, Sherzod Turaev, Nazar Zaki. "Abnormal Head Movements in Neurological Conditions: A Knowledge-Based Dataset with Application to Cervical Dystonia". arxiv. 2026-04-03T00:00:00.000Z. URL: https://arxiv.org/abs/2604.01962

[15] Li, X., Liu, J., Liu, J., Richardson, T. I., Dage, J. L., Klein, R. C., Chen, D. Z., Huang, K., Zhang, J., Johnson, T. S., Shi, Y.. "ARSA: an autonomous research scientist for target nomination in Alzheimer's disease and related dementias". bioRxiv. 2026-09-24T00:00:00.000Z. URL: https://doi.org/10.64898/2026.09.19.752836

[16] Dmitrichenko, O., Baldizzi, G., Schmidt, T., Bussy, A., Colliot, O., Lutti, A., Kherif, F., Draganski, B.. "The quest for the best: manual, atlas- and spatial prior-based delineation of locus coeruleus". medRxiv. 2025-11-25T00:00:00.000Z. URL: https://doi.org/10.1101/2025.11.24.25340859

[17] Quah, S. K. L., Jo, B., Geniesse, C., Uddin, L. Q., Mumford, J. A., Barch, D. M., Fair, D. A., Gotlib, I. H., Poldrack, R. A., Saggar, M.. "A Data-Driven Latent Variable Approach to Validating the Research Domain Criteria Framework". bioRxiv. 2024-11-08T00:00:00.000Z. URL: https://doi.org/10.1101/2024.01.31.577486

[18] Mikkel Vinding. "A unified framework for reliability analysis in neuroimaging with Krippendorff’s α". psyarxiv. 2025-01-20T16:54:39.888Z. URL: https://doi.org/10.1002/ima.70192

[19] Peraza, J. A., Kent, J. D., Blair, R. W., Poline, J.-B., Nichols, T. E., de la Vega, A., Laird, A. R.. "Advancing image-based meta-analysis for fMRI: A framework for leveraging NeuroVault data". bioRxiv. 2025-03-10T00:00:00.000Z. URL: https://doi.org/10.1101/2025.03.06.641922

[20] Jimenez-Marin, A., Boulanger, S., Tellaetxe-Elorriaga, I., Escudero, I., Cabrera-Zubizarreta, A., De Sousa, G., Freijo, M., Tejada, P. I., Toro, R., Diez, I., Erramuzpe, A., Cortes, J. M.. "COGNET-STROKE: A NOVEL BRAIN DYSCONNECTIVITY TOOL FOR PREDICTION OF COGNITIVE DEFICITS AFTER STROKE". medRxiv. 2026-05-14T00:00:00.000Z. URL: https://doi.org/10.1101/2025.08.13.25333589

[21] Bakker, L., Caganek, T., Rooprai, A., Hume, S.. "Systematic reviews in minutes to hours using artificial intelligence". medRxiv. 2026-02-10T00:00:00.000Z. URL: https://doi.org/10.64898/2026.02.06.26345764

[22] Cai, X., Geng, Y., Du, Y., Westerman, B., Wang, D., Ma, C., Vallejo, J. J. G.. "Utilizing ChatGPT to select literature for meta-analysis shows workload reduction while maintaining a similar recall level as manual curation". medRxiv. 2025-01-09T00:00:00.000Z. URL: https://doi.org/10.1101/2023.09.06.23295072

[23] Najpande, G., Titiya, P., Khan, M. A., Khakwani, K. Z. R., Sonbol, M. B., Riaz, I. B., Gupta, V.. "Reliable extraction of clinical-trial evidence with provenance, verification, and targeted review". bioRxiv. 2026-10-08T00:00:00.000Z. URL: https://doi.org/10.64898/2026.10.01.756091

[24] Delporte, M., Tamimi, R., Mehta, S., Choi, E., Zhang, Y., Shi, Y.. "MetaFemina: development and evaluation of a large language model-assisted platform for automated meta-analysis of nutritional exposures and breast, ovarian, and uterine cancer risk". medRxiv. 2026-08-21T00:00:00.000Z. URL: https://doi.org/10.64898/2026.08.18.26360713

How this survey was made

This survey answers one question: “How have automated and semi-automated text-mining pipelines been validated against manual curation when extracting brain region–function associations from the neuroscience literature?”

We keep our own collection of recent preprints and refresh it every day from bioRxiv, medRxiv, arXiv, PsyArXiv, SocArXiv and ChemRxiv. For each survey, we search that collection for recent work and keep the strongest matches: papers that use the same terms as the question, and papers on the same topic in different words.

The text was written from the titles and abstracts of those papers; the full texts were not read. Statements that draw on a particular paper are marked with a number in brackets. The reference list just above is the 24 papers whose work this text draws on, chosen from the 137 papers those searches returned; the rest of the shortlist is not printed. It is not a complete review of everything published on the question.

Treat this as a starting point and a reading list, not a peer-reviewed review. The searches behind this survey ran on October 22, 2026 and covered work from the previous two years.