LLMs Remain Sensitive to Problem Representation in Math, Even with Code-Based Reasoning
A new arXiv preprint finds that large language models (LLMs) often change their answers' correctness when mathematically equivalent problems are presented in different surface forms, such as story, symbolic, or equation formats. Introducing code-based reasoning, where models generate and execute Python code, improves performance for some LLMs but does not fully resolve this sensitivity; errors instead shift to issues like protocol violations or code execution failures. The study highlights that even subtle changes in problem wording can significantly affect LLM reliability.
Why it matters: This work suggests that the way problems are presented remains a critical factor in LLM reliability for mathematical tasks, with implications for evaluation and deployment in real-world applications.
Full story at: arXiv AI/ML ↗