Spaghetti Architect: A Contamination-Resistant, Multi-Language Code Dataset Generator
Researchers present Spaghetti Architect, a tool that generates code datasets in five programming languages (Python, JavaScript, Go, Java, C++) with controlled difficulty and resistance to data contamination. The tool produces correct-by-construction programs from a known-optimal reference, allowing for reliable evaluation of code models. It enables fine-grained control over code 'messiness' and labels each instance along intrinsic and incidental difficulty axes.
Why it matters: This tool addresses the persistent challenge of data contamination in code model evaluation by enabling the creation of fresh, controlled, and contamination-resistant datasets.
Full story at: arXiv Software Engineering ↗