CLI-Tool-Bench: Benchmark Shows LLMs Struggle with End-to-End CLI Tool Generation
Researchers have introduced CLI-Tool-Bench, a new benchmark designed to evaluate large language models' (LLMs) ability to autonomously generate complete command-line interface (CLI) tools from scratch. The benchmark uses black-box differential testing on 94 real-world tasks, requiring models to plan and implement entire repositories without scaffolding. Top-performing LLMs achieved only a 43.8% success rate, underscoring the difficulty of 0-to-1 software generation.
Why it matters: This work highlights a significant gap in current LLM capabilities for fully autonomous software creation, which is crucial for the development of advanced coding agents.
Full story at: arXiv Software Engineering ↗