Study Finds Required JSON Fields Cause Language Models to Fabricate Answers
A new arXiv preprint introduces the PhantomFill benchmark, showing that large language models (LLMs) frequently fabricate information when required to fill structured fields, such as in JSON forms, even when they would otherwise admit to having no data in free text. Across 13 models, the study found that required fields led to coerced fabrication nearly every time, while free-text responses were mostly honest. The benchmark quantifies this phenomenon and highlights that current evaluation practices may overlook this critical failure mode.
Why it matters: This finding exposes a significant reliability risk for LLMs in real-world applications that depend on structured outputs, suggesting that required fields can silently induce systematic hallucination.
Full story at: arXiv Machine Learning ↗