LLMs Show Instability When Answering Paraphrased Questions, Study Finds
A new arXiv preprint reports that large language models (LLMs) often provide inconsistent answers when the same question is rephrased, with mismatch rates exceeding 23% across several benchmarks. While overall accuracy remains relatively stable, the study finds that single-prompt correctness does not reliably indicate model reliability. The authors also demonstrate that prompting models with self-generated paraphrases can help recover correct answers that might otherwise be missed.
Why it matters: The findings highlight a significant limitation in current LLM evaluation practices, suggesting that accuracy alone may not reflect true model reliability for real-world use.
Full story at: arXiv AI/ML ↗