Explanation Effects Are Conditional: What Reader Studies Actually Measure When They Test Saliency Maps

Question: How do human-AI interaction studies in medical imaging evaluate whether saliency or Grad-CAM explanations change clinician diagnostic accuracy, and which design factors determine when explanations help versus mislead?

Answer

Taken across these papers, the literature does not so much answer the question as reveal that it is rarely posed as stated. The dominant evaluation of saliency methods in medical imaging remains mathematical — perturbation-based faithfulness and localization overlap — and that evaluation is itself unstable, with method rankings shifting under perturbation strategy and box-extraction rules, and gradient-weighted maps barely clearing position-null controls [1] [11]. Where humans are studied, the outcome is more often preference, trust, or acceptance than clinician accuracy, and the few studies that do measure accuracy find the explanation's presence to be a weak predictor next to three design factors: stratification by AI correctness and reader expertise, matching of explanation type to the reader's trust state and prior, and the timing and granularity with which advice is delivered [6] [9] [14]. Explanations mislead specifically when they raise the probability of following the AI without raising the reader's ability to discriminate correct from incorrect advice [25]. This survey's position is therefore that the field's evidence supports explanation-as-conditional-intervention, not explanation-as-additive-benefit, and that studies reporting a single unstratified mean accuracy delta cannot distinguish helping from misleading.

Most medical-imaging saliency research never measures a clinician's accuracy

Saliency evaluation in medical imaging is routinely conducted as a benchmarking exercise among attribution methods — one breast MRI study compares Last-layer Attention, Attention Rollout, Grad-SAM, Gradient Attention Rollout, GMAR, Grad-CAM and HiResCAM without any human participants [1]. Within that paradigm, conclusions are design-dependent rather than robust: method rankings vary across intensity-based perturbations and transformer attention masking, and benchmarking requires separating class-specific from class-agnostic explanations, with Grad-CAM and Gradient Attention Rollout strongest among class-specific methods but their relative order still sensitive to protocol [1]. The spatial target itself is shakier than commonly assumed, since class activation mapping and its gradient-weighted variant beat a position-null control on only two to four of seven annotated findings on ChestX-ray14 and two of eight on CheXlocalize, while three board-certified radiologists clear the same null on four to seven, and the extraction threshold, results regime and mask-to-box rule each move a result by more than most reported method effects [11]. When clinicians are consulted, the instrument is frequently a rating rather than a test: 33 physicians rated a bounding box plus report combination above other tested XAI methods on understandability, completeness, speed and applicability, and half trusted false AI diagnoses across all tested methods [19]. Even where attributions align with the classifier, they may carry little information for domain experts, as shown in a speech-based Parkinson's detection study [3]. Preregistered studies that measure subjective, objective and mathematical evaluation families in the same participants are rare, and where done the families disagree [18].

Effects are stratified by AI correctness and reader expertise, so unstratified deltas are uninterpretable

Explanation effects in the retrieved work are conditional on who is reading and whether the AI is right. In a two-experiment study of 623 lay people and 153 primary care physicians with a skin-tone-balanced diagnostic model, AI assistance improved accuracy and reduced disparities, but LLM explanations produced divergent effects: lay users showed greater automation bias — accuracy boosted when the AI was correct and reduced when it erred — while experienced physicians benefited irrespective of AI accuracy [6]. Among 28 pathology experts estimating tumour cell percentages, AI support improved performance yet produced a 7% automation bias rate, defined as previously correct independent judgments overturned by incorrect advice, with professional experience and self-efficacy associated with lower dependence on the system [15]. A scoping review of 45 studies concluded that explainable AI increased clinician trust but could also increase trust in incorrect recommendations, potentially harming collaborative performance, and that expert clinicians may benefit less from AI decision support than non-experts, though with mixed results [16]. Studies of prediction-level assistance with no explanation show why decomposition matters: AI assistance raised pooled sensitivity for critical abnormalities by 6.9 percentage points while lowering specificity by 5.5 points, with no significant change in confidence [2]. The theoretical reason to expect any benefit at all is information or capability asymmetry between human and model, the two identified sources of complementarity, which is also why complementarity is rarely observed [22].

Timing, advice granularity and trust-state matching set the sign of the effect

The strongest evidence that explanation format is a manipulable cause rather than a fixed attribute comes from interventions that change when and in what form the explanation arrives. Providing supporting explanations when user trust is low and counter-explanations when trust is high yielded up to a 38% reduction in inappropriate reliance and a 20% improvement in decision accuracy, with forced pauses inserted adaptively to reduce over-reliance [9]. Presentation order acts in the opposite direction: showing the AI suggestion first led to worse outcomes when the AI was incorrect, for both lay users and physicians [6]. Granularity matters non-monotonically — among 228 critical care physicians, those given simple model recommendations chose optimal treatments in 56.3% of decisions versus 42.9% for a less prescriptive advice type, and the simple-recommendation group overcame poor advice [14]. Cognitive load amplifies anchoring without changing the frequency of automation bias: time strain did not increase how often automation bias occurred but intensified its severity, and anchoring on system output increased under time pressure [15]. Bundled reader-side cues can move performance — a randomized trial with 72 physicians found a dual-component nudge combining a benchmark-accuracy anchoring cue and a case-specific selective-attention cue improved rubric-scored diagnostic reasoning by 7.6 percentage points [13]. Retrospective feedback is a separate lever with a distinct risk: unidirectional feedback on overrides influenced reflection-on-action but risked reducing reflection-in-action, with opposing shifts in reliance leaving overall team performance unchanged [24].

Plausibility and mathematical faithfulness do not map monotonically onto understanding

Visually plausible explanations need not reflect the model's decision process and may therefore mislead clinicians, which is the stated motivation for perturbation-based faithfulness work [1]. Manipulating explanation correctness at four known levels (100%, 85%, 70%, 55%) showed the relationship is not smooth: forward-simulation accuracy dropped at 70% and 55% relative to fully correct explanations, with no further reduction below 70% and no conclusive difference between 85% and 100%, and even fully correct explanations did not guarantee understanding, with accuracy bimodal across participants [12]. Mathematical faithfulness metrics were only partially related to user performance, and sometimes counterintuitively [18]. A more fundamental problem is that a faithful map may faithfully report a shortcut: mechanistic auditing of ViT-Base 16 identified a single attention head functioning as an annotation reader that attended to corner text rather than lung pathology in 62% of 200 COVID-19 images, replicating across two datasets and three architectures, surviving full fine-tuning, and remaining invisible at output level, since single-head and multi-head ablation yielded p = 0.340 and p = 0.532 [4]. Downstream, heatmap-based explanations were not associated with public judgments of physician responsibility, complaint likelihood or litigation intent in a 652-participant randomized vignette experiment, with no explanation-by-source interaction [5].

Accuracy deltas conceal the failure mode; reliance-based endpoints are the correct dependent variable

A single accuracy change cannot separate following the AI more often from distinguishing correct from incorrect advice, and a decision-theoretic formulation defines reliance as the probability of following the recommendation and separates it from the ability to differentiate signals, allowing loss from mis-reliance to be distinguished from loss from poor discrimination [25]. A corresponding measurement framework organises evaluation into outcomes, reliance behaviour, safety signals and learning over time, and argues for interaction traces rather than self-reported trust, precisely because failures arise from overuse when the AI is wrong and underuse when it is helpful [21]. Both failure directions appear in the retrieved data: in a cooperative question-answering study, humans missed 3.9% of correct AI suggestions while over-relying on misleading ones in 1.7% of cases, reported model confidence was near chance when human and AI disagreed, and confirmation bias drove 64.5% under-reliance when the AI agreed with an initially incorrect human answer [23]. Subjective instruments are a poor substitute: participants reported no differences in trust or satisfaction across LIME, Grad-CAM and Guided Backpropagation despite differences in objective performance, while mathematical metrics favoured a different method than the one that helped users [18]. Reviews of the interaction literature similarly identify a fluency trap that inflates trust without improving decisions, and conclude that static interfaces and miscalibrated trust limit AI efficacy because performance hinges on aligning transparency with cognitive workflow rather than maximising explanation [7]; trust across users spans from critical to over-reliant, so visualisations must be designed to support healthy skepticism as well as trust [10].

Conflicts and gaps

Two direct conflicts remain unresolved. First, the spatial quality of gradient-weighted maps is contested: one study ranks Grad-CAM as a strongest class-specific method [1], while another finds CAM-family readouts barely exceed position-null controls and that protocol choices outweigh method effects and box area alone can reverse a paired comparison [11]. Second, whether explanations change outcomes at all depends on which outcome is chosen: Grad-CAM improved user performance in one preregistered saliency study [18], heatmap explanations had no detectable effect on a vignette-based judgment [5], and explanation provision sometimes showed no clear benefit or was worse than none [17]. The central gap is design-level: no retrieved study isolates a Grad-CAM or saliency overlay as the randomised factor with clinician diagnostic accuracy as the primary endpoint and prespecified stratification by AI correctness, reader expertise and case difficulty. The closest design — an interactive application allowing radiologists to engage model heatmaps while forming diagnoses, compared against image-only reading — reports its methods but no effect estimate in the retrieved abstract [8]. Most studies manipulate explanation presence or format rather than explanation validity; the one study that manipulates validity at known levels does so in a synthetic time-series task where domain knowledge cannot substitute for the explanation [12]. Prevalence is a further unaddressed confound: readers overestimated the positive predictive value of an aneurysm tool at 62.9% against a simulation-based 15.4%, and a three-minute base-rate education reduced acceptance of false positives from 22.5% to 12.7% [20] — meaning explanation effects measured at convenience-case prevalence may not transfer, yet none of the retrieved explanation studies reports reader base-rate beliefs as a covariate.

What would change the answer

A preregistered multireader multicase study with explanation format randomised within reader — no overlay, Grad-CAM overlay, bounding box, bounding box plus report — with a washout interval, reader-level sensitivity and specificity as coprimary endpoints, and prespecified stratification by AI correctness, expertise and case difficulty would settle whether format is causal. Existing designs supply every component: the washout crossover structure [2], the stratified automation-bias analysis [15], and the base-rate manipulation [20]. Adding reliance-decomposition endpoints would separate discrimination gains from mere agreement [25] [21]. To isolate explanation validity from recommendation content, the decisive manipulation is pairing the same prediction with a faithful versus a deliberately scrambled or mismatched attribution while holding AI correctness fixed — the clinical analogue of graded explanation-correctness levels [12]. Because the localization benchmark against which heatmap quality is judged currently has no held-out validation set and was used to make the very design choices it scores [11], a released, radiologist-validated localization set with position-null controls would be a prerequisite for treating spatial accuracy as a mediator. Finally, mechanistic pre-screening by attention and ablation audits before any reader is shown an explanation would prevent reader studies from testing explanations of a spurious shortcut that output-level metrics cannot detect [4], and joint collection of subjective, objective and mathematical measures in the same participants would expose where those families diverge [18].

References

[1] Peachapong Poolpol, Henrik H. J. Detjen, Eike Petersen. "Faithful Faithfulness Evaluations: Challenges & Pitfalls Learned from a Breast MRI Case Study". arxiv. 2026-09-23T00:00:00.000Z. URL: https://arxiv.org/abs/2609.25978

[2] Novak, A., Shah, R., Espinosa Morgado, A. T., Robert, D., Kumar, S., Oke, J., Bhatia, K., Romsauerova, A., Das, T., Narbone, M., Dharmadhikari, R., Harrison, M., Vimalesvaran, K., Gooch, J., Woznitza, N., Lowe, D. J., Shuaib, H., Ather, S., AI-REACT Reader Study Group,. "Artificial Intelligence-assisted reader evaluation in acute CT head interpretation (AI-REACT): a multireader multicase study.". medRxiv. 2025-10-17T00:00:00.000Z. URL: https://doi.org/10.1101/2025.10.17.25337471

[3] Eleonora Mancini, Francesco Paissan, Paolo Torroni, Mirco Ravanelli, Cem Subakan. "Investigating the Effectiveness of Explainability Methods in Parkinson's Detection from Speech". arxiv. 2024-11-14T00:00:00.000Z. URL: https://arxiv.org/abs/2411.08013

[4] Joshua Muhumuza, Cornelius Kalenzi. "Spurious Attention Heads in Vision Transformers: Mechanistic Identification of Annotation-Driven Bias in Medical Chest X-Ray Analysis". arxiv. 2026-10-16T00:00:00.000Z. URL: https://arxiv.org/abs/2610.21638

[5] Choi, J., Kim, Y. J., Lyu, P., Luan, Y. L., Toh, S. M.. "Explainable AI and public reactions to AI-involved adverse diagnostic events: a vignette study". medRxiv. 2026-06-02T00:00:00.000Z. URL: https://doi.org/10.64898/2026.05.26.26353870

[6] Xuhai Xu, Haoyu Hu, Haoran Zhang, Will Ke Wang, Reina Wang, Luis R. Soenksen, Omar Badri, Sheharbano Jafry, Elise Burger, Lotanna Nwandu, Apoorva Mehta, Erik P. Duhaime, Asif Qasim, Hause Lin, Janis Pereira, Jonathan Hershon, Paulius Mui, Alejandro A. Gru, Noémie Elhadad, Lena Mamykina, Matthew Groh, Philipp Tschandl, Roxana Daneshjou, Marzyeh Ghassemi. "Explainable AI as a Double-Edged Sword in Dermatology: The Impact on Clinicians versus The Public". arxiv. 2025-12-16T00:00:00.000Z. URL: https://doi.org/2512.125

[7] Most. Sharmin Sultana Samu, Nafisa Khan, Kazi Toufique Elahi, Tasnuva Binte Rahman, Md. Rakibul Islam, Farig Sadeque. "AI as Teammate or Tool? A Review of Human-AI Interaction in Decision Support". arxiv. 2026-02-19T00:00:00.000Z. URL: https://arxiv.org/abs/2602.15865

[8] Dack, E., Dai, C., Hoppe, H., Krueselmann, P., Meiler, S., Jutidamrongphan, W., Wang, L., Tang, K.. "Towards Interpretable AI Second Opinions: Foundation Model Heatmaps in Radiology". medRxiv. 2026-08-23T00:00:00.000Z. URL: https://doi.org/10.64898/2026.08.20.26360908

[9] Tejas Srinivasan, Jesse Thomason. "Adjust for Trust: Mitigating Trust-Induced Inappropriate Reliance on AI Assistance". arxiv. 2026-01-27T00:00:00.000Z. URL: https://arxiv.org/abs/2502.13321

[10] G. Elisabeta Marai, Marc Baaden, Michael Behrisch, Michael Krone, Pere-Pau Vázquez. "Healthy skepticism in AI: a data visualization research agenda". arxiv. 2026-10-08T00:00:00.000Z. URL: https://arxiv.org/abs/2610.0974

[11] Shixin Qiu. "Rethinking weakly supervised localization on chest radiographs: position nulls, protocol dependence, and PACE". arxiv. 2026-10-20T00:00:00.000Z. URL: https://arxiv.org/abs/2610.23902

[12] Gregor Baer, Chao Zhang, Isel Grau, Pieter Van Gorp. "Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding". arxiv. 2026-08-12T00:00:00.000Z. URL: https://arxiv.org/abs/2603.25251

[13] Qazi, I. A., Ali, A., Khawaja, A. U., Akhtar, M. J., Sheikh, A. Z., Alizai, M. H.. "Mitigating Automation Bias in Physician-LLM Diagnostic Reasoning Using Behavioral Nudges: A Randomized Controlled Trial". medRxiv. 2026-06-05T00:00:00.000Z. URL: https://doi.org/10.64898/2026.06.01.26354596

[14] Venkatesh Sivaraman, Joel Levin, Eric P. Mason, Victor Talisa, Suchi Saria, Jean Feng, Andrew J. King, Jeremy M. Kahn, Adam Perer. "Indirect Influences of Artificial Intelligence Advice on Complex Sepsis Treatment Decisions". arxiv. 2026-10-20T00:00:00.000Z. URL: https://arxiv.org/abs/2610.24313

[15] Emely Rosbach, Jonas Ammeling, Jonathan Ganz, Christof Albert Bertram, Thomas Conrad, Andreas Riener, Marc Aubreville. "Stuck on Suggestions: Automation Bias, the Anchoring Effect, and the Factors That Shape Them in Computational Pathology". arxiv. 2026-09-29T00:00:00.000Z. URL: https://doi.org/10.59275/j.melba.2026-87b1

[16] Jackson, N. J., Brown, K. E., Miller, R., Murrow, M., Cauley, M., Collins, B. X., Novak, L. L., Benda, N., Ancker, J. S.. "Factors Influencing the Effectiveness of AI-Assisted Decision-Making in Medicine: A Scoping Review". medRxiv. 2026-01-07T00:00:00.000Z. URL: https://doi.org/10.1101/2025.09.02.25334863

[17] Michael Winikoff. "Contrastive explanations of BDI agents". arxiv. 2026-02-17T00:00:00.000Z. URL: https://arxiv.org/abs/2602.13323

[18] Felix Kares, Timo Speith, Hanwei Zhang, Markus Langer. "Are explainable AI (XAI) evaluation strategies aligned? Comparing subjective, objective, and mathematical evaluation measures using saliency maps". arxiv. 2026-09-15T00:00:00.000Z. URL: https://arxiv.org/abs/2504.17023

[19] Julia Wagner, Tim Schlippe. "A User-Centric Analysis of Explainability in AI-Based Medical Image Diagnosis". arxiv. 2026-05-06T00:00:00.000Z. URL: https://arxiv.org/abs/2605.02903

[20] Kim, S. H., Le Guellec, B., Rossmueller, P., Schramm, S., Boese, L., Nikoubashman, O., Kottlors, J., Lichtenstein, T., Strotzer, Q., Meddeb, A., Ziegelmeyer, S., Steinhelfer, L., Prucker, P., Berberich, C., Canisius, J., Kreutzinger, V., Hartl, F., Schmitzer, L., Rosenkranz, E., Leonhardt, Y., Beutel, T.-M., Bitzer, F., Maegerlein, C., Boeckh-Behrens, T., Baum, T., Makowski, M. R., Kirschke, J. S., Bressem, K. K., Adams, L. C., Baird, G. L., Wiestler, B., Hedderich, D. M.. "A 3-Minute Education on the False Positive Paradox Improves Trust Calibration in AI-Assisted Intracranial Aneurysm Detection: A Multinational Randomized Controlled Reader Study". medRxiv. 2026-08-28T00:00:00.000Z. URL: https://doi.org/10.64898/2026.08.25.26361324

[21] Min Hun Lee. "From Accuracy to Readiness: Metrics and Benchmarks for Human-AI Decision-Making". arxiv. 2026-03-20T00:00:00.000Z. URL: https://doi.org/10.1145/3772363.3798377

[22] Patrick Hemmer, Max Schemmer, Niklas Kühl, Michael Vössing, Gerhard Satzger. "Complementarity in Human-AI Collaboration: Concept, Sources, and Evidence". arxiv. 2024-11-27T00:00:00.000Z. URL: https://arxiv.org/abs/2404.00029

[23] Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig, Irene Ying, Tianyi Zhou, Jordan Boyd-Graber. "AI, Take the Wheel: What Drives Delegation and Trust in Human-Computer Cooperative Question Answering?". arxiv. 2026-05-28T00:00:00.000Z. URL: https://arxiv.org/abs/2605.28255

[24] Joshua Holstein, Philipp Spitzer, Sebastian Schäfer, Maximilian Förster, Gerhard Satzger. "Think Once, Reflect Twice: How Feedback Shapes Human-AI Collaboration". arxiv. 2026-10-15T00:00:00.000Z. URL: https://arxiv.org/abs/2610.19991

[25] Ziyang Guo, Yifan Wu, Jason Hartline, Jessica Hullman. "A Decision Theoretic Framework for Measuring AI Reliance". arxiv. 2025-09-26T00:00:00.000Z. URL: https://arxiv.org/abs/2401.15356

How this survey was made

This survey answers one question: “How do human-AI interaction studies in medical imaging evaluate whether saliency or Grad-CAM explanations actually change clinician diagnostic accuracy, and what design factors determine when explanations help versus mislead?”

We keep our own collection of recent preprints and refresh it every day from bioRxiv, medRxiv, arXiv, PsyArXiv, SocArXiv and ChemRxiv. For each survey, we search that collection for recent work and keep the strongest matches: papers that use the same terms as the question, and papers on the same topic in different words.

The text was written from the titles and abstracts of those papers; the full texts were not read. Statements that draw on a particular paper are marked with a number in brackets. The reference list just above is the 25 papers whose work this text draws on, chosen from the 155 papers those searches returned; the rest of the shortlist is not printed. It is not a complete review of everything published on the question.

Treat this as a starting point and a reading list, not a peer-reviewed review. The searches behind this survey ran on October 22, 2026 and covered work from the previous two years.