Study Finds Persistent Contamination Risks in Dynamic Benchmarks for Multimodal Fact-Checking
A recent arXiv preprint reports that dynamic benchmarks, intended to prevent contamination in multimodal automated fact-checking, still exhibit significant contamination rates, with 17–29% of post-cut-off claims potentially affected. The study demonstrates that such contamination can inflate evaluation metrics and alter system rankings, challenging the assumption that dynamic benchmarks are inherently contamination-free.
Why it matters: This finding raises concerns about the reliability of current evaluation practices for fact-checking systems and suggests the need for stricter contamination controls.
Full story at: arXiv Computation and Language ↗