Calibrated Selective Fact-Checking via Evidence Chain Evaluation
Researchers introduce Evidence Chain Evaluation (ECE), a selective fact-checking framework for large language models (LLMs) that allows the system to abstain from issuing a verdict when supporting evidence is weak or inconsistent. On the ECE-Bench benchmark, ECE achieves 91.6% standard accuracy, 93.7% coverage, and 97.8% selective accuracy on claims it chooses to answer, deferring 6 out of 95 cases, mostly in low-reliability evidence settings. While ECE does not outperform the strongest retrieval baseline on aggregate calibration metrics, it demonstrates a strong trade-off between accuracy and selective abstention.
Why it matters: This work provides a practical safety mechanism for LLM-based fact-checking by enabling abstention on uncertain claims, addressing a key reliability challenge in automated verification systems.
Full story at: arXiv AI/ML ↗