Work Saved and Recall Are Produced by Different Mechanisms, and Only One of Them Is Ever Guaranteed

Question: How do active learning and statistical stopping rules for screening prioritization in systematic reviews compare in work saved and recall guarantees across simulation and benchmark studies?

Answer

Across the retrieved papers, active learning and statistical stopping rules are not competing options that can be ranked on a single efficiency–safety frontier; they do different work. Active learning (and ranking generally) is what generates the work saved: it beats random screening in every simulated scenario tested across more than 29,000 runs [4], and AL-assisted simulation studies routinely report 71–93.6% reductions in records screened [7] or screening burdens near 11% of the corpus [8]. Statistical stopping rules are what convert that prioritization into a defensible halting decision, and the one paper that compares them directly finds that a fixed score threshold fell below 95% recall in 3 of 28 Cochrane reviews, whereas ranking with statistical stopping reached 95% recall in all 51 reviews — but only by reading 86–88% of records [3]. The honest reading of this literature is that the trade-off is not "AL versus stopping rules" but "effort versus the strength of the recall claim": high work saving is routinely demonstrated, recall guarantees are not, and every reported recall figure is a realized outcome on retrospectively labelled data rather than a bound that would hold under a stopping rule chosen in advance. The papers that combine the two levers — AL ranking plus a statistical stopping criterion — obtain both the best work-saving figures and the least fragile recall behaviour [3].

Active learning reliably buys work saved, but the size of the gain is dataset- and model-dependent rather than a fixed property of the method

Large-scale simulation supports a robust ordering rather than a magnitude: in two simulations totalling over 29,000 runs, active learning outperformed random screening in every scenario tested, with gains ranging from considerable to near-flawless depending on dataset, model, and screening stage [4]. Model choice at fixed recall target moves the burden substantially — SVM with TF-IDF needed only 6.87–7.66% of records at 95–99.5% recall, and SVM with Doc2Vec achieved WSS@100 of 88.5%, while sentence-BERT features degraded severely at 100% recall and required screening over 65% of the corpus [8]. Integration with human decision-making preserves the advantage: a semi-automated AL system reached 98.2–100% recall at its defined stopping point with workload savings of 71.0–93.6% relative to human-only screening [7]. Reported work saved at a nominal 95% recall spans 46.3% to 89.3% across five public datasets with 0.5–2.0% prevalence, i.e. the same metric varies by nearly a factor of two across corpora [14].

Recall at the stopping point is a realized outcome on labelled data, not a guarantee, and it degrades exactly where screening is hardest

The strongest evidence that recall is not protected comes from the largest and most heterogeneous dataset in a parameter sweep: AL-assisted screening achieved 100% recall of full-text inclusions with 89–90% reduction in records screened on three of four datasets, but only 87% recall on the 16,218-record dataset with 0.6% title/abstract and 0.08% full-text inclusions [2]. Calibration targets are met approximately, not exactly: a classifier calibrated for 99% recall achieved 98.6% recall across eight validation reviews [11]. Human screening itself carries a high false-exclusion error rate of 10.89% (95% CI 4.43%–17.35%), which matters because recall measured against an imperfect gold standard cannot certify what that standard missed [10]. The scoping review of 60 simulation studies found uniformly positive recommendations for AL but concluded that the field needs standardized metrics and open data before its efficiency claims can be compared across studies [1].

Statistical stopping rules change the failure mode from silent recall loss to visible effort, which is the correct trade for a systematic review

Where a threshold is set on the score itself, the recall contract is broken quietly: a fixed score threshold dropped below 95% recall in 3 of 28 Cochrane reviews, while ranking with statistical stopping reached 95% recall in all 51 reviews at a cost of reading 86–88% of records [3]. This reframes the guarantee as effort-bounded rather than free — the criterion does not create information, it forces the reviewer to keep reading until the evidence supports stopping, and the authors' own conclusion is that stopping safely required screening decisions rather than label-free scores [3]. The same asymmetry appears outside AL: a sensitivity-and-precision-maximizing RCT filter reduced overall number-needed-to-screen by 83% while retrieving 95.9% of includes, against 98.2% for the sensitivity-maximizing filter [9]. Workload-saving estimates collapse when recall must be calibrated per review rather than per corpus — 12.5% reduction in reviews versus 80% for database curation with the same 99%-recall-calibrated classifier [11] — and the AL literature itself identifies the absence of a generalisable stopping criterion as the main reason efficiency gains remain modest [7]. Tutorial guidance accordingly treats stopping-rule selection as a sensitivity–efficiency calibration problem rather than a default [6].

The reported comparisons are not commensurable, and much of the apparent disagreement is measurement, not method

Recall figures across systems are not comparable because the methods have different objectives, because inter-assessor disagreement is unresolved, and because each represents a different recall–precision and cost–benefit trade-off — a point made explicitly in the eDiscovery context, where continuous active learning trained on 2,000 reviewer judgments was argued to be state-of-the-art in effectiveness and efficiency [5]. Ground truth differs between studies: some score against full-text inclusion decisions [7] while others score against title/abstract labels or curated corpora [8], and still others use database-scale automated inclusion where the acceptance criterion is database-level recall rather than review-level [11]. Precision at the stopping point can be as low as 2% while recall is 94–100% [14], so papers reporting "efficiency" without both axes are reporting different quantities.

Conflicts and gaps

The clearest conflict is between the simulation consensus and the outlier result: AL outperformed random screening in every one of more than 29,000 simulated runs [4], and all 60 reviewed studies recommended it [1], yet a systematic parameter sweep found recall falling to 87% on the hardest of four datasets under the same recommended procedure [2]. A second conflict concerns magnitude: AL studies report 71–93.6% workload savings [7] while a calibrated-threshold classifier reports 12.5% for review-level use [11] — a gap too large to be attributable to model quality alone and probably reflecting different denominators and different recall contracts. The decisive gap is that no retrieved paper compares an active learning prioritization policy against a statistical stopping rule on a common benchmark with the stopping rule fixed before screening: the one direct comparison is explicitly post hoc across 31 Cochrane reviews [3], and the AL system with the best reported workload savings states that prospective testing is still required [7]. Human–AI collaborative screening at scale shows that post-processing and noisy-label filtering are load-bearing components of the workflow [16], which means "work saved" often includes downstream cleaning that no stopping rule accounts for.

What would change the answer

A pre-registered benchmark that freezes the stopping rule before screening and scores realized recall and realized effort on the same held-out reviews would settle the central question; the held-out-then-post-hoc structure already used at scale [3] provides the template, and the proposal to separate method development from data-generating-mechanism development in living synthetic benchmarks supplies the neutrality mechanism [13]. Importing anytime-valid inference would give stopping decisions error control regardless of when screening halts [12], and active, anytime-valid risk-controlling prediction sets already formalize the combination of a label budget with a sequential risk guarantee, which is precisely the missing object — a recall guarantee conditioned on the stopping policy itself [15]. A prospective randomized comparison of AL-plus-statistical-stopping against manual screening, as the JARVIS authors call for [7], would supply the one number simulations cannot: work saved under a guarantee that was never allowed to be chosen after the fact.

References

[1] Jelle Teijema, Guilherme Ribeiro, Sofie Seuren, Daniel Anadria, Ayoub Bagheri, Rens van de Schoot. "Simulation-based Active Learning for Systematic Reviews: A Scoping Review of Literature". psyarxiv. 2025-04-25T08:49:49.557Z. URL: https://osf.io/67zmt_v2

[2] Verdonschot, R. G., Dilles, T., Cahir, C., De Graef, M., Holis, R. V., Frydenlund, J., Denig, P., Grimes, T., Karapinar-Carkit, F. G., Schor, M.. "To Include or Not to Include? A prescription from the pharmacy on how to use active learning assisted screening in systematic reviews". medRxiv. 2025-09-28T00:00:00.000Z. URL: https://doi.org/10.1101/2025.09.26.25336705

[3] Jo, T., Lee, D., Lee, L.. "Label-free, low-cost AI ranking for systematic review screening". medRxiv. 2026-10-19T00:00:00.000Z. URL: https://doi.org/10.64898/2026.10.09.26365173

[4] Jelle Teijema, Jonathan de Bruin, Ayoub Bagheri, Rens van de Schoot. "Large-Scale Simulation Study of Active Learning Models for Systematic Reviews". psyarxiv. 2025-03-03T14:45:58.848Z. URL: https://osf.io/2w3rm_v2

[5] Gordon V. Cormack, Maura R. Grossman, Jeremy Pickens. "Comparing GenAI to Machine Learning and Manual Review for eDiscovery". arxiv. 2026-10-14T00:00:00.000Z. URL: https://arxiv.org/abs/2610.16925

[6] Tim Fütterer, Lars König, Diego Campos, Ronny Scherer, Steffen Zitzmann, Martin Hecht. "Fast-Track Your Abstract Screening: Mastering ASReview for Accelerating Abstract Screening and Evaluating Decisions From Automatic Screening Methods". psyarxiv. 2026-03-28T17:25:41.974Z. URL: https://osf.io/bwkt5_v1

[7] Barreto, G. H. C., Burke, C., Davies, P., Halicka, M., Paterson, C., Swinton, P., Saunders, B., Higgins, J. P. T.. "JARVIS, should this study be selected for full-text screening? Performance of a Joint AI-ReViewer Interactive Screening tool for systematic reviews". medRxiv. 2026-04-11T00:00:00.000Z. URL: https://doi.org/10.64898/2026.04.08.26350384

[8] Abdelkader, W., Xie, D., Lokker, C., Chu, L., Schandelmaier, S., Saha, A., Afzal, M., Iorio, A.. "A machine learning model to support the screening for methods guidance articles in MEDLINE: A performance evaluation of ASReview simulation mode". medRxiv. 2025-10-15T00:00:00.000Z. URL: https://doi.org/10.1101/2025.10.13.25337935

[9] Fulbright, H. A., Marshall, D., Evans, C., Corbett, M.. "Does the sensitivity- and precision-maximizing RCT filter find all 'included' records retrieved by the sensitivity-maximizing filter on Ovid MEDLINE? An investigation using 14 Cochrane reviews". medRxiv. 2026-03-23T00:00:00.000Z. URL: https://doi.org/10.64898/2026.03.20.26348876

[10] Rutger Neeleman, Berke Yazan, Emily Westerbeek, Wouter van Ballegooijen, Rens van de Schoot. "Updated Approach to Error Rates in Systematic Review Screening: Integrating Active Learning, Large Language Models, and Full-Text Screening Data". psyarxiv. 2026-01-26T14:56:22.053Z. URL: https://osf.io/r3m2d_v1

[11] Kavita Kothari, Anna Noel-Storr, James Thomas. "Machine Learning Reduced Workload for COVID-19 Literature". socarxiv. 2025-11-26T07:00:00.102Z. URL: https://osf.io/gu3bt_v1

[12] Alexander Ly, Udo Boehm, Peter Grünwald, Aaditya Ramdas, Don van Ravenzwaaij. "A Tutorial on Safe Anytime-Valid Inference: Practical Maximally Flexible Sampling Designs for Experiments Based on e-Values". psyarxiv. 2025-10-06T17:39:15.413Z. URL: https://osf.io/h5vae_v3

[13] František Bartoš, Samuel Pawel, Björn S. Siepe. "Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies". arxiv. 2026-10-15T00:00:00.000Z. URL: https://arxiv.org/abs/2510.19489

[14] Yuki Kataoka, Masahiro Banno, Michihito Kyo, Shuri Nakao, Tomoo Sato, Shunsuke Taito, Tomohiro Takayama, Takahiro Tsuge, Yasushi Tsujimoto, Ryuhei So, Toshi A. Furukawa. "TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Study Selection in Systematic Reviews". arxiv. 2026-09-24T00:00:00.000Z. URL: https://arxiv.org/abs/2604.08602

[15] Ziyu Xu, Nikos Karampatziakis, Paul Mineiro. "Active, anytime-valid risk controlling prediction sets". arxiv. 2024-11-01T00:00:00.000Z. URL: https://arxiv.org/abs/2406.1049

[16] Jonas Bergmann, Tiago Azzi, Rutger Neeleman, Kianush Monschau, Elena Jalsovec, Emily Westerbeek, Felix Weijdema, Jonathan de Bruin, Qixiang Fang, Rens van de Schoot. "Screenathon 2.0: Human–AI Collaborative Screening Applied to Patient-Generated Health Data". psyarxiv. 2026-01-09T09:41:08.889Z. URL: https://osf.io/e2qfj_v1

How this survey was made

This survey answers one question: “How do active learning and statistical stopping rules for screening prioritization in systematic reviews compare in work saved and recall guarantees across simulation and benchmark studies?”

We keep our own collection of recent preprints and refresh it every day from bioRxiv, medRxiv, arXiv, PsyArXiv, SocArXiv and ChemRxiv. For each survey, we search that collection for recent work and keep the strongest matches: papers that use the same terms as the question, and papers on the same topic in different words.

The text was written from the titles and abstracts of those papers; the full texts were not read. Statements that draw on a particular paper are marked with a number in brackets. The reference list just above is the 16 papers whose work this text draws on, chosen from the 121 papers those searches returned; the rest of the shortlist is not printed. It is not a complete review of everything published on the question.

Treat this as a starting point and a reading list, not a peer-reviewed review. The searches behind this survey ran on October 21, 2026 and covered work from the previous two years.