CIGPO: Contextual Information-Gain Policy Optimization Prevents GRPO Collapse in Multi-Turn Evidence-Reading Agents
A new method called CIGPO introduces per-turn rewards based on information gain to address a failure mode in outcome-only GRPO training for multi-turn evidence-reading agents. In experiments on HotpotQA with Qwen2.5-3B-Instruct, CIGPO achieved a standard F1 score of 0.518, outperforming GRPO's best checkpoint (0.430) and avoiding the collapse to format-violating outputs observed in GRPO. The approach maintains reward variance and prevents optimization deadlock caused by zero-advantage lock-in.
Why it matters: This work provides a practical solution to a concrete optimization failure in reinforcement learning for LLM agents, enabling more stable and reliable training of multi-turn evidence-reading systems.
Full story at: arXiv Machine Learning ↗