LLMs Struggle to Detect Faulty Code When Documentation Remains Intact, Study Finds
A new arXiv preprint introduces TRACE, a method for evaluating how large language models (LLMs) allocate trust across conflicting software artifacts such as code, documentation, and tests. Testing seven LLMs on Java method bundles with injected faults, the study finds that models are much better at detecting errors in documentation than in code, and often fail to deprioritize faulty implementations when documentation appears correct. The results suggest LLMs may over-rely on documentation and miss subtle code errors, with confidence scores offering little help in distinguishing correct from incorrect judgments.
Why it matters: This highlights a significant limitation in current LLM-based coding assistants, raising concerns for their use in safety- or correctness-critical software development.
Full story at: arXiv Software Engineering ↗