Controlled Study Shows How Pretraining Influences RL Gains in LLM Reasoning Tasks
A new preprint uses chess as a controlled environment to systematically investigate how pretraining choices affect reinforcement learning (RL) outcomes for reasoning in large language models (LLMs). The study finds that post-RL performance is strongly predicted by pretraining loss, and that RL amplifies correct moves on easy puzzles while surfacing new correct moves on harder ones. These patterns also appear in math-domain text tasks, suggesting generalizable principles for the pretraining-to-RL pipeline.
Why it matters: This work offers a quantitative framework for understanding how pretraining impacts the effectiveness of RL in improving LLM reasoning, informing future model development strategies.
Full story at: arXiv Computation and Language ↗