Recall Without Precision: How Region–Function Mining Pipelines Have Actually Been Validated

Question: How have automated and semi-automated text-mining pipelines been validated against expert-curated meta-analyses when extracting brain region–function associations from heterogeneous neuroimaging literature?

Answer

Taken together, the retrieved papers show that this validation has almost never been done in the form the question presumes. The single case in this corpus that measures a mining pipeline against an expert systematic review reports a recall-oriented operating point — 98% recall at 36% precision, with one target missed out of all targets across three seed structures, while finding three times as many targets as two person-weeks of manual curation [1]. Everything more recent validates against proxies: agreement with reference brain maps from the Human Connectome Project rather than with curated region–function assignments [4], or superior performance against baseline models inside a shared embedding space [3] [7]. The expert-curated meta-analysis that would serve as criterion is meanwhile itself methodologically uneven [8] and, on at least one influential case, produces region labels that are artefacts of thresholded averaging [6]. Across these papers, then, the field has validated pipelines for finding literature and for predicting maps, but not for reproducing curated region–function knowledge.

The one benchmark against a systematic review buys recall by sacrificing precision

The only head-to-head comparison in this corpus set text-mining output against a systematic literature review of the projections of three subcortical structures, in rat and primate including human, and found that mining located three times as many targets as two person-weeks of curation, at 98% recall but 36% precision, missing only one target overall [1]. That design is recall-first by construction: the pipeline is scored on whether the curated target set is recovered, not on whether what it adds beyond that set is correct. The same paper cross-checked the mined targets against probabilistic tractography output rather than against further expert curation, so the added targets were never adjudicated by the reference standard [1]. A 36% precision ceiling is the operative result for anyone planning to substitute such a pipeline for curation: roughly two-thirds of extracted associations are not in the curated set, and the retrieved literature contains no follow-up that adjudicates whether those are errors or discoveries.

Contemporary pipelines validate against brain maps and baselines, not against curated associations

The most rigorous recent validation framework in this corpus concerns image selection rather than text extraction: three manual image-based meta-analyses were compared against Human Connectome Project reference maps, with manual selection most precise and heuristic selection a robust alternative for heterogeneous domains such as emotion processing, and robust estimators required for heterogeneous task sets [4]. That is a template for how a text-driven selection step could be validated, and it is explicitly framed as agreement with reference maps, not with expert-curated region–function labels [4]. Text-to-brain models follow the same pattern: a hyperbolic brain–text framework claims superiority over keyword-retrieval and linear-mapping baselines without an external curated criterion [3], and a contrastive text-to-brain framework reports improved retrieval and map reconstruction and claims to infer activations in regions discussed but not coordinate-reported, again measured against baselines and reconstruction quality [7]. A CLIP-style text-to-brain model evaluated against HCP group-level activation maps across emotion, language and motor domains found performance optimised with full-text articles rather than abstracts and with a curated cognitive ontology of task–concept–domain mappings — but it degraded on noisy subject-level maps, which is a statement about map prediction, not about extraction fidelity [9].

The expert meta-analysis that would serve as criterion is itself unvalidated

Any validation against expert-curated meta-analyses inherits their weaknesses, and a large-scale audit of over 14,000 papers found that only 20% of neuroimaging meta-analyses were pre-registered, only half listed their exact search strategy, 68% included fewer than 30 studies and thus were not robust to publication bias, and there is likely location bias in which papers are included [8]. More pointedly, a four-study investigation concluded that "language regions" are artefacts of averaging heterogeneous representations: those regions behave as connectivity hubs coordinating periphery activity that averages out after thresholding, and linguistic representations are decodable throughout the brain when specific representations rather than central tendency are examined [6]. A pipeline trained to reproduce expert meta-analytic maps therefore risks reproducing a thresholding artefact and would be scored as accurate for doing so. The validation target is not a fixed truth but a methodological choice.

Where experts are used, they function as glossaries and annotation codes rather than as meta-analytic benchmarks

The most explicit expert-anchored validation in this corpus uses a manually defined glossary of thirty-six concepts grouped into three hypotheses, against which a council of ten local language models scores studies, producing auditable disagreement measures rather than accuracy against a meta-analysis [10]. Validation in adjacent extraction tasks follows the same annotation-code pattern: LLM annotation of ion channel and receptor subtype metadata from 5,133 model files was scored against 1,100 manually annotated files with a 253-file held-out test set, reaching 96.0% accuracy and F1 0.962 at type level and 88.1% accuracy with F1 0.878 at subtype level [12]. Literature-mining platforms that project onto standard brains are described as extracting ontologically defined entities and their associations with anatomy from PubMed via SCAIView, with association strength computed by entropy and a transformation matrix linking over 800 anatomical terms to parcel coordinates, but the abstract reports no accuracy against curated associations [2]. The expert-curated alternative is built entirely by hand: 106 neocortical and 18 cerebellar fields identified from 1,465 human studies through manual analysis of published figures and manual alignment to templates, with inclusion requiring convergent functional evidence and lesion support [5] — exactly the resource that could anchor a benchmark, and not reported as one.

Conflicts and gaps

The largest conflict is between the 36% precision reported when mining is scored against a systematic review [1] and the near-ceiling performance reported when LLM screening is scored against benchmark meta-analyses, where one tool identified all 163 benchmark-included studies at 100% sensitivity and 99.4% pooled specificity across 270,626 abstracts and reduced workload by 91.7% [16], and pooled screening sensitivity 0.92 and specificity 0.94 across 18 studies [15]. These do not measure the same thing: binary eligibility screening is far easier than extracting the correct region, the correct function term, and the correct directional association from a heterogeneous report, and no retrieved paper reports precision and recall for that specific extraction step against a curated meta-analysis. Reliability of the extraction layer is a second unaddressed gap: two nominally identical LLM screening runs agreed on 91.7% of records yet differed on 94 records including 29 verified eligible ones, and no workflow recovered all eligible records [13], while human screening itself misclassifies at a weighted 3.51% with false exclusions at 10.89% [11] — meaning any benchmark's reference labels carry their own error, and the human ceiling is unmeasured for region–function extraction. Whether full text beats abstracts is unresolved across systems favouring full text [9] and platforms restricted to PubMed [2], and whether domain fine-tuning helps conflicts between a finding that domain-specific models perform numerically similarly to base models [9] and a benchmark on which a neuroscience-tuned model outperformed base models and experts at predicting results [14].

What would change the answer

A held-out design would settle it: take expert-curated meta-analyses with published inclusion lists, exclude their included studies from the mining corpus, and score mined region–function pairs against the curated pairs with precision, recall and F1 — extending the manual-versus-heuristic-versus-reference-map design already used for image selection [4] from maps to associations. Dual independent human annotation on a stratified heterogeneous sample, with kappa and a held-out split as in repository metadata annotation [12], would establish the human ceiling the current benchmarks lack. Run-to-run and version-to-version replication of the pipeline [13] would separate model capability from workflow variance. Finally, benchmarks must include cases where the curated meta-analysis is expected to be wrong — regions whose apparent function is an averaging artefact [6] — and should add prospective prediction of unseen results as an orthogonal criterion independent of the training literature [14].

References

[1] Xavier Vasques, Renaud Richardet, Sean L Hill, David Slater, Jean-Cedric Chappelier, Etienne Pralong, Jocelyne Bloch, Bogdan Draganski, Laura Cif. "Automatic target validation based on neuroscientific literature mining for tractography". arxiv. 2025-03-05T00:00:00.000Z. URL: https://doi.org/10.3389/fnana.2015.00066

[2] Stefanovski, L., Bülau, K., Martin, L., Huettl, C., Langford, C., Palmer, J., Sacks, M., Deger, L., Pille, M., Schirner, M., Meier, J., Neudorfer, C., Horn, A., Solodkin, A., Thirion, B., Hofmann-Apitius, M., Jacobs, M., Tom Kodamullil, A., for the Alzheimer's Disease Neuroimaging Initiative,, Ritter, P.. "An Interactive Brain Atlas of Knowledge". bioRxiv. 2026-01-26T00:00:00.000Z. URL: https://doi.org/10.64898/2025.12.12.694043

[3] Seunghun Baek, Jaejin Lee, Jaeyoon Sim, Minjae Jeong, Won Hwa Kim. "MNM : Multi-level Neuroimaging Meta-analysis with Hyperbolic Brain-Text Representations". arxiv. 2025-11-27T00:00:00.000Z. URL: https://arxiv.org/abs/2511.21092

[4] Peraza, J. A., Kent, J. D., Blair, R. W., Poline, J.-B., Nichols, T. E., de la Vega, A., Laird, A. R.. "Advancing image-based meta-analysis for fMRI: A framework for leveraging NeuroVault data". bioRxiv. 2025-03-10T00:00:00.000Z. URL: https://doi.org/10.1101/2025.03.06.641922

[5] Poliva, O.. "The Brain Encyclopedia Atlas Project (BEAP): A Literature-Synthesis-Derived Functional Atlas of the Human Brain". bioRxiv. 2026-05-26T00:00:00.000Z. URL: https://doi.org/10.64898/2026.05.05.722843

[6] Aliko, S., Franch, M., Kewenig, V., Wang, B., Cooper, G. J., Glotfelty, A., Hayden, B., Small, S. L., Skipper, J. I.. "The entire brain, more or less, is at work: 'Language regions' are artefacts of averaging". bioRxiv. 2025-07-23T00:00:00.000Z. URL: https://doi.org/10.1101/2023.09.01.555886

[7] Ghayem, F., Meudec, R., Dockes, J., Thirion, B., Wassermann, D.. "NeuroConText: Contrastive Learning for Neuroscience Meta-Analysis with Rich Text Representation". bioRxiv. 2026-02-05T00:00:00.000Z. URL: https://doi.org/10.1101/2025.05.23.655707

[8] Oudyk, K. M., Dockes, J., Peraza, J. A., Kent, J., Torabi, M., Wang, M., McPherson, B. C., Mirhakimi, N., de la Vega, A., Laird, A., Poline, J.-B.. "Meta all the way down: An overview of neuroimaging meta-analyses". bioRxiv. 2025-01-22T00:00:00.000Z. URL: https://doi.org/10.1101/2025.01.21.634094

[9] Peraza, J. A., Kent, J. D., Nichols, T. E., Poline, J.-B., de la Vega, A., Laird, A. R.. "NiCLIP: Neuroimaging contrastive language-image pretraining model for predicting text from brain activation images". bioRxiv. 2026-07-11T00:00:00.000Z. URL: https://doi.org/10.1101/2025.06.14.659706

[10] Hamed Nejat, Alexander Maier, Jesse Spencer-Smith, André M. Bastos. "Ontology-constrained multi-LLM scoring of hypothesis support in the predictive processing literature". arxiv. 2026-06-05T00:00:00.000Z. URL: https://arxiv.org/abs/2606.05206

[11] Rutger Neeleman, Berke Yazan, Emily Westerbeek, Wouter van Ballegooijen, Rens van de Schoot. "Updated Approach to Error Rates in Systematic Review Screening: Integrating Active Learning, Large Language Models, and Full-Text Screening Data". psyarxiv. 2026-01-26T14:56:22.053Z. URL: https://osf.io/r3m2d_v1

[12] Cohen, I., Yu, H., McDougal, R. A.. "All Models are Wrong, Some are Annotated: Automating Metadata in Biomedical Repositories". bioRxiv. 2026-04-27T00:00:00.000Z. URL: https://doi.org/10.64898/2026.04.23.720371

[13] Nikol Figalová, Lynn Huestegge, Anne Böckler-Raettig. "Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency". arxiv. 2026-08-28T00:00:00.000Z. URL: https://arxiv.org/abs/2608.26885

[14] Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K. Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O. Cohen, Valentina Borghesani, Anton Pashkov, Daniele Marinazzo, Jonathan Nicholas, Alessandro Salatiello, Ilia Sucholutsky, Pasquale Minervini, Sepehr Razavi, Roberta Rocca, Elkhan Yusifov, Tereza Okalova, Nianlong Gu, Martin Ferianc, Mikail Khona, Kaustubh R. Patil, Pui-Shee Lee, Rui Mata, Nicholas E. Myers, Jennifer K Bizley, Sebastian Musslick, Isil Poyraz Bilgin, Guiomar Niso, Justin M. Ales, Michael Gaebler, N Apurva Ratan Murty, Leyla Loued-Khenissi, Anna Behler, Chloe M. Hall, Jessica Dafflon, Sherry Dongqi Bao, Bradley C. Love. "Large language models surpass human experts in predicting neuroscience results". arxiv. 2024-12-02T00:00:00.000Z. URL: https://doi.org/10.1038/s41562-024-02046-9

[15] Chenggong, X., Weichang, K., Liuting, P., Diaoxin, Q., Yuxuan, Y., Bin, W., Liang, H.. "Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta-analysis". medRxiv. 2026-03-19T00:00:00.000Z. URL: https://doi.org/10.64898/2026.03.17.26348656

[16] Teperikidis, L., Polymenakis, K., Trampoukis, C.. "Validation of Synthesa AI, a Large Language Model-Based Screening Tool for Systematic Reviews: Results from Nine Studies". medRxiv. 2025-07-17T00:00:00.000Z. URL: https://doi.org/10.1101/2025.07.16.25331632

How this survey was made

This survey answers one question: “How have automated and semi-automated text-mining pipelines been validated against expert-curated meta-analyses when extracting brain region–function associations from heterogeneous neuroimaging literature?”

We keep our own collection of recent preprints and refresh it every day from bioRxiv, medRxiv, arXiv, PsyArXiv, SocArXiv and ChemRxiv. For each survey, we search that collection for recent work and keep the strongest matches: papers that use the same terms as the question, and papers on the same topic in different words.

The text was written from the titles and abstracts of those papers; the full texts were not read. Statements that draw on a particular paper are marked with a number in brackets. The reference list just above is the 16 papers whose work this text draws on, chosen from the 165 papers those searches returned; the rest of the shortlist is not printed. It is not a complete review of everything published on the question.

Treat this as a starting point and a reading list, not a peer-reviewed review. The searches behind this survey ran on October 21, 2026 and covered work from the previous two years.