Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
A new theory and method for verifying optimization models generated by large language models (LLMs) is introduced, using 'falsification-based' test batteries that guarantee zero false positives. The approach tests models through solver calls on systematically altered problem instances, without needing reference models. Experiments on 326 ground-truth models show the method detects 70% of certain mutant errors and identifies 40.4% of errors missed by standard execution-accuracy scoring, with a 0% false-positive rate.
Why it matters: This work offers a principled and reliable way to catch silent failures in LLM-generated optimization models, addressing a key risk in automated decision-problem translation.
Full story at: arXiv Software Engineering ↗