A new preprint introduces the Energy-guided Recursive Model (ERM), which applies Hopfield energy functions to select among candidate solutions during recursive reasoning. ERM achieves state-of-the-art accuracy on structured tasks, including 98.97% on Sudoku, 88.04% on PPBench, and 99.30% on Maze, outperforming previous models. The method provides a principled inference mechanism without relying on additional q-heads or heuristic voting.
Why it matters: This work demonstrates a novel, principled approach to inference in recursive reasoning models, potentially advancing performance on complex structured problems.
Researchers present a theoretical framework explaining how large language models (LLMs) can store factual knowledge in MLP layers at an information-theoretically optimal rate. They introduce a Transformer-compatible MLP construction that achieves optimal storage capacity scaling and requires 10–104 times fewer parameters than previous methods. The approach supports arbitrary input/output geometries and can be used within Transformer blocks for factual recall tasks. The work also demonstrates modular fact editing by swapping a Transformer's MLP with a new one.
Why it matters: This work advances understanding of how LLMs efficiently store factual knowledge and provides a practical method for more efficient and modular fact storage in Transformer architectures.
Researchers introduce a bilevel optimization framework for Direct Preference Optimization (DPO) that can recover clean-data performance even when preference labels are noisy. Their approach uses meta-learning without requiring metadata, leveraging central-difference approximation and LoRA for scalable training. Experiments on TL;DR summarization and Anthropic HH dialogue tasks demonstrate improved performance over standard DPO baselines across various noise rates.
Why it matters: This work addresses a key limitation of DPO by enabling robust alignment of language models in the presence of noisy preference data, which is common in real-world applications.
Researchers have developed conservation laws for diffusion models using generalized extrinsic information transfer (GEXIT) functions. Their work shows that the data–model cross-entropy can be exactly characterized as an integral of local information-theoretic derivatives along the noise path, providing a unified framework for both discrete and continuous diffusion models. This approach implies that training diffusion models reduces to learning marginal posteriors, and the theory is validated on synthetic data and benchmarks such as text8 and CIFAR-10.
Why it matters: This framework offers a unified theoretical understanding of diffusion model training, which could inform the development of more principled and efficient denoising objectives.
A new preprint finds that the widely used multiplicative repetition penalty in LLM inference engines (such as HuggingFace, vLLM, and llama.cpp) is ill-defined because it branches on the sign of raw logits, whose zero-point is arbitrary. This gauge dependence means that re-centering logits can change 58-96% of greedy tokens at a typical penalty setting and can severely degrade structured output, dropping JSON schema compliance from 97% to 23%. The study shows that applying the penalty to normalized log-probabilities instead of raw logits eliminates these issues.
Why it matters: This work exposes a fundamental flaw in a common LLM inference technique that can silently degrade output quality and reliability across many deployed systems.
A new arXiv preprint presents a method for detecting whether a large language model (LLM) was trained via distillation from another model, using reference-based membership inference. The approach compares a model and an earlier checkpoint from the same lineage to identify the likely teacher model, even when details of the distillation process are unknown. The method demonstrates near-perfect accuracy in both controlled experiments and real-world scenarios, and introduces statistical tests for teacher attribution and distillation detection. The study also provides new evidence of distillation relationships among models such as QwQ, DeepSeek-R1, and GPT-OSS.
Why it matters: This work provides a practical tool for detecting and attributing model distillation, addressing concerns about model provenance and compliance with usage policies.
A new method called Depth-Entropy Guided Sampling (DEGS) leverages layer-wise entropy collapse as a quality signal to enhance large language model (LLM) reasoning at test time, without requiring any additional training. DEGS integrates sequence likelihood with collapse depth in a Markov Chain Monte Carlo (MCMC) framework, achieving state-of-the-art accuracy among training-free methods on several reasoning benchmarks. The approach demonstrates particular strength out-of-domain and on challenging tasks, sometimes surpassing reinforcement learning (RL)-trained models, all with minimal computational overhead.
Why it matters: DEGS offers a practical, training-free alternative to RL for boosting LLM reasoning, potentially lowering the cost and complexity of deploying advanced reasoning systems.
Researchers have developed a foundation transformer model pretrained on multimodal sequences of user events for financial services. By unifying heterogeneous data sources into chronological event sequences and using next-event prediction, the model learns general-purpose representations that can be applied to multiple downstream tasks. The system outperformed traditional task-specific models and was deployed in production at a major Eastern European bank, where it led to measurable improvements in business metrics.
Why it matters: This work demonstrates a practical and effective foundation model for financial event sequences, showing real-world impact through improved predictive performance and reduced development overhead in a production banking environment.
A new preprint demonstrates that low-precision training can cause neural network weights to 'freeze'—stop updating—when gradient updates become too small to affect stored values, due to rounding. This freeze is deterministic and can be predicted in advance using only high-precision training trajectories and the mantissa length of the target precision. The phenomenon was observed in both small GPT models and a 124-million-parameter GPT-2, and stochastic rounding was shown to prevent the freeze.
Why it matters: This work reveals a predictable and silent failure mode in low-precision neural network training, with practical implications for the design and reliability of efficient AI systems.
A preprint study analyzes four open-weight transformer language models and finds that multimodal instruction-tuning causes a qualitative shift in how identity-specifying system prompts are encoded in hidden-state trajectories. Specifically, the encoding changes from being direction-based in the base model to magnitude-based after multimodal RLHF instruction-tuning, a reorganization not observed in RL distillation or SFT regimes. The study uses geometric metrics and permutation tests to support these findings.
Why it matters: This work uncovers a previously unrecognized effect of multimodal instruction-tuning on internal model representations, which could inform future approaches to model interpretability and control.
This study systematically compares Turbo-Quant and SpectralQuant KV-cache compression methods, using a statistical validation methodology to evaluate non-dominated schemes. The research finds that eigenbasis-based methods perform poorly on heavy-tailed data due to covariance instability but excel in structured regimes, with the effective semantic dimension adapting to calibration budgets rather than true data rank.
Why it matters: The work provides a rigorous framework for evaluating KV-cache compression techniques, which is important for optimizing memory and latency in large language model inference.
Researchers introduce an output-aware safety guardrail for multimodal large language models (MLLMs) that predicts unsafe generations from hidden states, rather than relying solely on input analysis. Their method achieves safety performance comparable to existing approaches but significantly reduces over-refusal, thereby maintaining the model's utility and leveraging its intrinsic safety mechanisms.
Why it matters: This work offers a practical advance in balancing safety and usability for MLLMs by enabling more targeted safety interventions and reducing unnecessary refusals.
Google researchers introduce Quota Marketplace, a market-based system for allocating ML training chips such as GPUs using dynamic pricing to accommodate workloads with varying values. The system, implemented and deployed at Google, is designed to achieve Pareto efficiency and max-min fairness, and to better align resource allocation with organizational priorities. The paper details both the theoretical foundations and practical deployment, including metrics demonstrating its impact.
Why it matters: Efficient allocation of scarce ML training resources is a major challenge for organizations, and this work demonstrates a practical, scalable solution that addresses heterogeneous workload values.
A new preprint introduces Low-Rank Attention Residuals (LR-AttnRes), a method that uses low-dimensional keys for depthwise routing in large language models (LLMs) while retaining full-dimensional residual values. This approach decouples routing from representation, leading to improved validation loss and reduced computational cost. The authors present two variants—Projected and Sliced LR-AttnRes—and release code and models for further research.
Why it matters: The work demonstrates that effective depthwise routing in LLMs can be achieved with significantly fewer dimensions, suggesting a path toward more efficient model architectures.
EvoClawBench is a new benchmark designed to test whether AI agents can transform evidence from their own runs into reusable skills that improve future executions. The benchmark covers 100 tasks across coding, data, office, security, operations, and domain-document workflows, and supports multiple agent runtimes. Experiments with OpenClaw and nanobot show that the ability to learn reusable skills is selective and cost-sensitive: some models improve with skill learning, while others experience performance drops or collapse. This demonstrates that skill learning from experience is not an automatic benefit for agent systems.
Why it matters: This work isolates and rigorously tests a critical capability for autonomous agents—learning from their own experience—and shows that it is not guaranteed, informing the design of more robust agent systems.
A new preprint introduces SPARC, a spectral-algebraic theory that formally explains why autoregressive language models (LLMs) often fail to correct their own errors but can fix identical errors when attributed to external sources. The theory proves that this 'self-correction blind spot' arises when the spectral radius of the error-propagation operator is at least one, and it derives a precise activation threshold for correction markers. Experiments across four model backbones and a visual autoregressive probe validate the theory, with predictions matching observed blind-spot rates within 3.2% RMSE.
Why it matters: This work provides the first formal, quantitative model of the self-correction blind spot in autoregressive generation, offering new insights that could improve the reliability of LLMs and related models.
A new preprint evaluates Scientific Machine Learning (SciML) methods for macroeconomic forecasting and finds that structural priors can act as misregularizers when they do not align with the true data-generating process. In tests across 23 countries, less-constrained models like ARIMA and Neural ODEs consistently outperformed more-constrained models such as PINNs and UDEs. The study highlights the challenges of low-frequency macroeconomic prediction and cautions that more structural assumptions do not always yield better results.
Why it matters: This work challenges the assumption that adding structural priors always improves model performance, offering important guidance for SciML practitioners.