VulWeaver is a novel approach that combines large language models (LLMs) with structured program analysis to improve vulnerability detection in source code. It achieves F1-scores of 0.76 on Java and 0.78 on C/C++ datasets, outperforming existing baselines by 17-25%. In real-world tests, VulWeaver detected 26 true vulnerabilities across 9 Java projects, with 15 confirmed by developers and 5 CVEs assigned.
Why it matters: VulWeaver demonstrates that integrating LLMs with program analysis can significantly advance automated vulnerability detection, with practical impact validated by developer confirmations and CVE assignments.
Researchers introduce PROBE, a benchmark framework for evaluating code generation by large language models (LLMs) across multiple dimensions: functional correctness, proximity to valid solutions, and code quality. PROBE assesses models using five programming languages and three prompting strategies, and includes analysis of common errors. The results show that LLMs struggle with more difficult problems and that smaller models perform worse on low-resource languages, often making fundamental errors.
Why it matters: PROBE enables a more thorough and systematic evaluation of LLM-generated code, revealing persistent reliability challenges that are not captured by existing benchmarks.
SeedSmith is an agentic LLM pipeline designed to generate initial seed inputs for directed fuzzing by emulating a security analyst's workflow. It iteratively explores codebases, resolves indirect calls, identifies crash preconditions, and synthesizes concrete inputs, serving as a fuzzer-agnostic seed generation front-end. In evaluations, SeedSmith enabled fuzzers to achieve crash-time speedups of 11.51x to 14.66x on Magma and to trigger 16 previously unreachable bugs across 10 projects with diverse input formats.
Why it matters: SeedSmith demonstrates a significant advance in automating and improving the efficiency of vulnerability discovery through LLM-driven seed generation for fuzzing.
A controlled experiment at a large Brazilian enterprise found that design-system-aware AI assistance reduced front-end development time by 46.7% to 69.4% across Angular, iOS, and Android stacks. The study also observed increased task completeness and reduced performance variability compared to manual or design-system-only development. Analysis indicated that AI tools helped reduce workflow friction and improved the consistency of design implementation.
Why it matters: This study provides empirical evidence that integrating AI with design systems can substantially improve the speed, consistency, and quality of industrial front-end development workflows.
Researchers introduce FairCoder, a benchmark that frames high-stakes decision-making as coding tasks to systematically probe large language model (LLM) bias in domains such as employment, education, and healthcare. The study also proposes FairScore, a new metric that accounts for both refusal behavior and group-level outcome diversity. Experiments on leading LLMs reveal consistent and previously underexplored bias patterns, including a tendency to prioritize applicants from high-income families in college admissions. The work provides a comprehensive framework for evaluating LLM bias in practical decision-making scenarios.
Why it matters: This research highlights the risks of deploying LLMs in real-world decision-making and offers tools for more thorough bias evaluation.
A new framework enables LLM-based coding agents to persistently learn from human review feedback by codifying accepted review comments as behavioral rules, without requiring model retraining. Deployed across a 35+ service microservices platform, the rule set expanded from 5 to 18 behavioral rules, eliminating recurrence of previously ruled-against error classes. The system shifts human review focus from low-level correctness to higher-level design validation.
Why it matters: This approach demonstrates a practical method for coding agents to continuously improve and reduce repetitive errors in real-world codebases without retraining.
A new preprint proposes 'falsifiable release gates' for self-improving AI systems, requiring each new capability to pass a machine-verifiable acceptance suite before deployment. The methodology is demonstrated on the Antahkarana open runtime, using seven gates to ensure safety invariants are maintained. The approach is open-sourced for reproducibility and is designed to be adaptable to other agent frameworks.
Why it matters: This work introduces a concrete, reproducible method for verifying safety in self-improving AI systems, addressing a key challenge in AI safety engineering.
Researchers have introduced XVada, a method for black-box context-free grammar inference that outperforms existing approaches like TreeVada in both accuracy and compactness. In empirical evaluations, XVada discovered a CVE in the Python Liquid engine and identified five additional bugs through fuzzing, all of which were subsequently fixed by the developers.
Why it matters: This work demonstrates a significant advance in automated grammar inference with practical impact, as evidenced by the discovery and remediation of real-world software vulnerabilities.
A study of 22 Triton and TileLang GPU kernels reveals that correctness-based evaluation can overlook severe performance issues—one TileLang LayerNorm kernel, for example, passes correctness checks but is over 300× slower than the PyTorch baseline. The underlying causes of inefficiency vary by kernel family, with some due to repairable authoring defects and others stemming from code-generation or autotuning limitations. The authors propose two lightweight screening methods—library-relative efficiency and roofline utilization—that successfully identify all functionally correct but inefficient kernels in their evaluation.
Why it matters: This work exposes a critical flaw in current GPU kernel evaluation practices and offers practical tools to ensure that functionally correct kernels also meet performance expectations.
Researchers have introduced PerfCodeBench, a benchmark designed to evaluate large language models (LLMs) on system-level high-performance code optimization tasks. PerfCodeBench features executable correctness checks, baseline implementations, and reference optimized solutions, enabling assessment of both code correctness and runtime efficiency. Results show a significant gap between model-generated code and expert-optimized solutions, particularly in tasks involving parallelism and GPU operations.
Why it matters: PerfCodeBench exposes the limitations of current LLMs in generating efficient, hardware-aware code for performance-critical systems, emphasizing the need for performance-oriented evaluation in code generation.
RESOURCE2SKILL is a framework that extracts and organizes executable skills for software agents from multimodal resources such as tutorial videos, code repositories, and articles. These skills are structured in a hierarchical Skill Wiki that integrates text, code, visual examples, and metadata, enabling agents to retrieve and compose relevant skills for complex tasks. In evaluations across seven authoring domains, RESOURCE2SKILL improved agent performance by an average of 11.9 percentage points compared to agents without skill libraries, and outperformed strong baselines in most tested scenarios.
Why it matters: This work demonstrates a significant advance in enabling software agents to autonomously acquire and utilize diverse, human-created resources as reusable skills, enhancing their adaptability and effectiveness in complex tasks.
Researchers introduce ArtifactGuide, a structured rubric for software artifact evaluation, and ArtifactCopilot, an LLM-based agent that automates the evaluation process. In tests on 60 real software artifacts, ArtifactCopilot achieved 70.56% badge-level agreement with human reviewers, outperforming baseline prompt-based approaches. A user study with 8 experienced researchers found that ArtifactCopilot's reports improved reviewer confidence and helped locate supporting evidence.
Why it matters: This work demonstrates a significant advance in automating software artifact evaluation, potentially reducing reviewer workload and improving consistency in the review process.
A new preprint introduces a vendor-neutral metric designed to evaluate whether evidence from agent-safety evaluations can reconstruct the decisions underlying safety claims. The metric assesses reconstructability across eight decision-property classes and includes a cross-harness adapter for generating Evidence Sufficiency Cards. Tests on public traces show sufficiency scores between 0.458 and 0.833, with replay preconditions unmet in all scored traces.
Why it matters: This work provides a standardized approach to evaluating the validity of agent safety claims, addressing a key challenge in building trustworthy AI systems.
A new method called ROBIN introduces white-box, head-level fairness debugging for transformer models by ranking attention heads based on their sensitivity to fairness probes and removing a small bias subspace from selected head outputs. In a four-model pilot study, ROBIN reduced the WinoBias gap while preserving language-modeling quality better than whole-head zeroing. The results indicate that both the selection of attention heads and the method of modification are important for effective bias repair.
Why it matters: This work demonstrates a targeted inference-time approach for mitigating bias in transformer models, offering a more precise alternative to retraining or coarse interventions.
A recent preprint demonstrates that providing feedback as line-anchored comments, using the FileMark VSCodium extension, can reduce the number of generated tokens by 22% to 58% compared to holistic prompts in AI code editing tasks. The study found that this approach also improves code correctness, with gains of 2 to 7 points for local models. The benefits are especially pronounced for longer files and for models with more room for improvement.
Why it matters: This work introduces a practical technique that can lower the cost and improve the accuracy of AI-assisted code editing, potentially enhancing developer workflows and model efficiency.
A new preprint introduces 'use-case-oriented regeneration,' a software sourcing paradigm that uses generative AI to synthesize only the specific dependency functionality required by a project, rather than relying on entire external libraries. In an evaluation across 180 repository-dependency pairs, the approach preserved 99.8% of observed behavior and reduced the exported API surface by 93%. This suggests that targeted code regeneration could be a feasible alternative to traditional software supply chains.
Why it matters: If widely adopted, this approach could significantly reduce software supply chain risks by enabling local verification of code rather than relying on external sources.
CT-Repair is an automated program repair framework that leverages Code Property Graphs (CPG) and Temporal Execution Graphs (TEG) to represent static and dynamic evidence. It uses three finite-state-machine-guided agents to analyze bugs from static, dynamic, and hybrid perspectives, and applies a filtering pipeline that reduces runtime evidence by over 94%. On 854 Java bugs from Defects4J v3.0, CT-Repair correctly repairs 489 bugs in a mixed-model configuration and 388 bugs under a controlled GPT-5.4-mini configuration, outperforming ReinFix and RepairAgent.
Why it matters: This work demonstrates that structured runtime evidence and multi-perspective reasoning can significantly improve automated program repair effectiveness without relying solely on larger patch-generation budgets.
A new method called E3 (Estimate, Execute, Expand) enables large language model (LLM) agents to estimate task difficulty before execution, reducing unnecessary context processing. On the MSE-Bench benchmark, E3 matches the strongest baseline's 100% task success rate while reducing costs by 85%, tokens by 91%, and inspected files by 92%. Live tests with gpt-4o on a real open-source library confirm that E3 remains leaner and faster than alternatives at comparable task success.
Why it matters: This work demonstrates a significant advance in reducing computational redundancy and operational costs for LLM agents, improving their scalability and efficiency.
A new method called EVITA uses multi-objective optimization to automatically generate driving scenarios that test interactions among multiple autonomous vehicles (AVs). Unlike traditional approaches that focus on single-AV testing, EVITA is designed to uncover safety-critical behaviors that emerge only when multiple AVs interact. Experimental results indicate that EVITA produces a greater diversity of AV interactions compared to existing scenario generation methods.
Why it matters: This approach addresses a key gap in AV safety testing by enabling the discovery of critical multi-vehicle interaction scenarios that single-vehicle tests may miss.
Researchers introduce Code-MUE, a black-box framework for measuring uncertainty in code large language models (LLMs) by analyzing execution-based semantic interaction graphs. The method quantifies semantic diversity using Von Neumann entropy and demonstrates a strong negative correlation with functional correctness (Spearman's up to -0.98) across eight models. Code-MUE outperforms lexical and embedding-based baselines for risk detection and selective prediction, addressing limitations of existing black-box uncertainty metrics for code.
Why it matters: This work provides a practical and effective approach to uncertainty estimation for closed-source code LLMs, supporting safer and more reliable automation in software engineering.