The Correctness Illusion in LLM-Generated GPU Kernels
A new preprint demonstrates that widely used benchmarks for evaluating LLM-generated GPU kernels can incorrectly certify buggy code as correct. The study shows that these benchmarks, which rely on fixed-shape, small-sample allclose checks, often miss subtle errors. By applying op-schema-aware seeded fuzzing with high-precision references, the authors catch all tested bugs without false positives across five different GPU classes.
Why it matters: This work reveals a critical flaw in current evaluation methods for LLM-generated GPU code, indicating that many kernels considered correct may actually harbor undetected bugs.
Full story at: arXiv Software Engineering ↗