Study Finds Attention Heatmaps in Medical VLMs Are Not Faithful Explanations
A new causal evaluation of attention and saliency heatmaps in medical Vision-Language Models (VLMs) for chest X-rays finds that none of the tested VLMs produce faithful visual explanations. The study assessed MedGemma, LLaVA-RAD, Qwen3-VL, and CheXagent, showing that their heatmaps either anti-correlate with causal importance or reflect near-text-only behavior. In contrast, standard chest X-ray classifiers passed all faithfulness metrics, indicating the issue is specific to VLM heatmaps.
Why it matters: This work challenges the reliability of widely used visual explanations in medical AI, showing that attention heatmaps can be visually reassuring but causally unfaithful, which has direct implications for clinical deployment and trust.
Full story at: arXiv Computer Vision ↗