A new preprint introduces NameRank, a metric designed to quantify how well large language models (LLMs) recognize specific entities, such as people and artifacts, from their training data. The study probes 4,685 entities across 36 models and finds that LLM recognition is primarily associated with named artifacts (like papers or tools) rather than credentials or titles. Additionally, institutional prestige is found to be a stronger predictor of recognition than citation counts.
Why it matters: This work offers a systematic approach to evaluating what LLMs 'know' about individuals and artifacts, informing discussions on fairness, transparency, and the design of retrieval-augmented AI systems.
A new preprint investigates citation faithfulness and coverage in a 4B-parameter on-device research agent. The study finds that increasing the amount of each source the model sees ('exposure') significantly improves citation faithfulness, while the ability to retrieve relevant sources ('retrieval recall') limits trustworthy coverage. The authors recommend first maximizing per-source exposure, then focusing on improving retrieval recall to further enhance citation quality.
Why it matters: This work offers practical guidance for improving the reliability of citations in small, on-device research agents by disentangling the roles of exposure and retrieval.
A new preprint introduces GenAI Evaluation, a governed and scalable pipeline for evaluating retail conversational agents in production. The system processes around 50,000 records daily and has evaluated over two million interactions, using schema-constrained LLM scoring to assess helpfulness, truthfulness, clarity, tone, and translation quality. Validation against nearly 13,000 human-labeled records showed a macro F1 score of 0.93 and 89% human-acceptability accuracy for translation tasks.
Why it matters: This work demonstrates a practical and reproducible framework for large-scale, multi-dimensional evaluation of conversational AI systems in real-world settings.
A new preprint critically examines the effectiveness of automatic harness evolution for LLM agents, finding that it does not consistently outperform simpler test-time scaling methods and often fails to generalize beyond public benchmarks. The study, using experiments on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, highlights the risk of overfitting and calls for more rigorous evaluation protocols and benchmarks. The authors provide their code for transparency and reproducibility.
Why it matters: This work questions the value of a widely used method for improving LLM agent performance and urges the community to adopt more robust evaluation standards.
Professor Devavrat Shah is designing methods that enable AI models to handle constant decision-making using limited computational resources. His efforts span both research and entrepreneurship to bridge the gap between AI and real-world applications.
Why it matters: This work could make AI systems more efficient and practical for real-world use.
MIT students designed, built, and tested a jet engine with the assistance of AI copilots, evaluating how AI can support the development of high-performance aerospace systems. The JARVIS Challenge investigated the effectiveness of AI in aiding complex engineering tasks.
Why it matters: This experiment highlights AI's potential to enhance and accelerate the design and manufacturing of advanced aerospace technologies.
Researchers introduce DAG-FM, a novel foundation model for causal discovery that decomposes the process into two auto-regressive stages using specialized Transformer-based modules. The model incorporates a Mixture-of-Leaf-Experts approach to adapt to diverse causal mechanisms and demonstrates state-of-the-art performance on both synthetic and real-world datasets, outperforming existing classical and foundation model approaches.
Why it matters: This work represents a significant advance in scalable causal discovery, enabling more accurate inference of cause-effect relationships from complex observational data.
A new preprint audits distributional reinforcement learning (RL) agents—specifically QR-DQN, C51, and IQN—and finds that 40-95% of their strongest risk trade-off claims are statistically refuted at 95% confidence. The study shows that the 'risk' learned by these agents is largely a training artifact, not a reflection of true environment stochasticity, and that following the agents' risk-based advice can sometimes lead to outcomes worse than random choice. The authors provide a statistical toolkit for auditing such claims and highlight pitfalls that can lead to misleading audit results.
Why it matters: This work raises significant concerns about the reliability of risk estimates from distributional RL agents, which are increasingly considered for safety-critical applications.
Researchers introduce WSqD, a new learning rate schedule that replaces the constant phase of the Warmup-Stable-Decay (WSD) schedule with a shifted inverse-square-root base, making it independent of the training horizon. The method achieves minimax-optimal convergence in stochastic convex optimization and, in language model pretraining experiments, matches or outperforms tuned WSD across different training horizons using a single peak learning rate.
Why it matters: WSqD could reduce the need for repeated learning rate tuning when extending training, potentially saving computational resources in large-scale model training.
A new preprint provides theoretical analysis showing that diffusion models such as DDIM and DDPM can efficiently adapt to data with unknown low-dimensional structure. The authors prove that, under exact score functions, the number of sampling iterations needed to achieve a given total variation distance scales with the intrinsic dimension of the data, rather than the ambient dimension. The results are extended to cases where the score function is learned from data, with kernel-based estimators shown to maintain this adaptivity under certain conditions.
Why it matters: This work offers the first rigorous theoretical evidence that diffusion models can efficiently sample from high-dimensional data with low intrinsic dimension, supporting and explaining their empirical success.
Researchers introduce MAD-Path, a new Markov Chain Monte Carlo (MCMC) method that samples from multimodal distributions by interpolating along diffusion paths rather than using traditional tempering. This approach preserves the relative weights of modes and improves mixing, with a Metropolis adjustment ensuring the target distribution remains invariant even with approximate intermediate scores. Experimental results demonstrate that MAD-Path achieves better exploration and more accurate mode-weight estimation compared to tempering-based MCMC methods.
Why it matters: MAD-Path addresses a longstanding challenge in statistical inference by providing a more reliable and principled method for sampling from complex, multimodal Bayesian posteriors.
A new preprint demonstrates that using role-aware summaries as code representations can improve file-level bug localization performance by up to 40% in Hit@5 compared to file-path representations, while reducing storage requirements by 10-20x relative to raw source code. The study also shows that integrating these representations into the Agentless pipeline achieves 94% Hit@6, outperforming the baseline. The results highlight the importance of code representation choices for balancing accuracy and computational cost in LLM-driven software debugging.
Why it matters: This work identifies a practical approach to enhance both efficiency and effectiveness in LLM-based bug localization, which could directly benefit software development workflows.
Researchers introduce BackendForge, a benchmark comprising 56 backend generation tasks derived from real open-source applications. The benchmark evaluates whether large language models (LLMs) can generate deployable and behaviorally correct backend services from specifications and OpenAPI contracts. The best-performing model, GPT-5.5, succeeded on 55.4% of tasks under a base oracle but only 28.6% under a more stringent final oracle, highlighting significant challenges for LLMs in producing complete backend services.
Why it matters: BackendForge exposes a substantial gap in current LLMs' ability to generate fully functional backend services, underscoring a key limitation for agentic coding applications.
Researchers introduce WebDesignIter, a framework that integrates a persistent knowledge graph (WebAppArchKG) to combine repository structure with explicit design knowledge for iterative front-end code generation. Evaluated on the Web-Bench benchmark, WebDesignIter achieves an average Pass@2 improvement of 9.55 percentage points over existing baselines and surpasses general-purpose coding agents such as Claude Code and SWE-Agent, while using fewer input tokens. Ablation studies indicate that the inclusion of design knowledge is the most significant factor in its performance gains.
Why it matters: This work shows that explicitly incorporating design knowledge into LLM-based coding agents can substantially improve the reliability and efficiency of complex, repository-level code generation tasks.
TerraRepair is a prototype large language model (LLM) agent that leverages tool grounding to repair Terraform misconfigurations identified by Infrastructure-as-Code (IaC) scanners. By retrieving dependency context, consulting provider schemas, and re-running scanners, TerraRepair significantly increases fix rates on AWS benchmarks—from 26.6% to 78.4% with Checkov and from 44.8% to 72.4% with Trivy—compared to a one-shot baseline. The agent also escalates issues when deployment-specific context is missing, reducing the risk of hallucinated fixes.
Why it matters: This work demonstrates that tool grounding can substantially improve the reliability and effectiveness of LLM-based automated repair for cloud infrastructure code, potentially reducing manual intervention.
A new preprint presents the first systematic empirical study of sycophancy bias in large language models (LLMs) used for code smell detection. The authors find that LLMs are highly sensitive to misleading prompts, with decision flip rates up to 72% and false alignment rates over 90%. They introduce Evidence-Guided Debiasing Prompting (EGDP), which reduces these rates to 12% and 21%, respectively, by enforcing evidence-first reasoning.
Why it matters: This work identifies a major reliability risk in LLM-based code analysis and demonstrates a practical method to mitigate sycophancy bias.
Motif is a system that passively observes browser activity to identify recurring interaction patterns and recommends automations to users. In a study with eight participants, Motif discovered more automatable patterns than users identified themselves, and most participants found the generated programs useful and relevant to their routines.
Why it matters: Motif represents a shift toward system-initiated automation, potentially making web workflow automation more accessible to non-technical users by removing the need for them to explicitly identify tasks to automate.
Researchers propose a framework that leverages historical real-bug mechanisms to generate synthetic bugs, which are then used as feedback to guide large language model (LLM)-based unit test generation. By constructing structural and semantic representations of bug records and retrieving relevant mechanisms, the system instantiates synthetic bugs as feedback targets for iterative test improvement. Evaluation on the Defects4J benchmark shows that this approach improves real-bug detection compared to several baseline methods.
Why it matters: This work demonstrates a novel and effective method for enhancing the ability of LLM-generated unit tests to detect real software bugs, potentially increasing software reliability.
A new preprint reports that model instability is a significant issue in software analytics, with repeated runs of the same optimizer agreeing on only 13.7% of test cases across 127 multi-objective software engineering optimization problems. The researchers show that by adjusting label allocation, model complexity, and scoring splits, agreement can be improved by a factor of 4.8 and the standard deviation of optimization error reduced by 22% on average, without degrading recommendation quality. The study argues that instability is not just noise but a measurable and manageable property, and recommends treating it as a standard evaluation axis.
Why it matters: This work provides a practical framework for measuring and managing model instability, which could enhance the reliability and trustworthiness of software engineering optimization tools.
A new framework called ACQUIRE is proposed to enhance LLM-based code repair by explicitly acquiring repository knowledge before generating patches. The system uses a Questioner-Answerer collaboration to identify and fill knowledge gaps, resulting in more accurate repairs. On the SWE-bench Verified benchmark, ACQUIRE improves Pass@1 accuracy by up to 4.4 percentage points compared to prior pre-repair methods, with only modest overhead.
Why it matters: This approach addresses a major source of factual errors in LLM coding agents by systematically improving their understanding of code repositories before attempting repairs.