A new preprint tests the Muon optimizer on low-rank matrix factorization, finding that it does not consistently outperform AdamW. The study suggests that Muon's previously reported advantages in large-scale deep learning may depend on factors such as scale, architecture, or hyperparameter sensitivity.
Why it matters: This work challenges assumptions about Muon's superiority and highlights the importance of controlled benchmarks for evaluating optimizers.
A new preprint introduces a FLOP-accounting framework for reinforcement learning (RL) post-training, breaking down compute usage into rollout/search, policy-update/learning, and reward-model evaluation. The study, using LoRA-adapted Qwen2.5 policies, finds that optimal allocation of compute resources depends on factors such as model size, compute budget, and reward system. The authors also propose RACE, a diagnostic protocol to help identify effective compute allocation regimes before committing to expensive validation runs.
Why it matters: This work offers a systematic approach to allocating limited post-training compute in RL, which is important for efficiently adapting foundation models.
A new preprint introduces Enlightenment, a training-free post-tuning method that enhances large-scale models by modifying shortcut connections rather than updating weights. The approach includes attention head-mixing for large language models and scalar-modulated residual connections for vision-language models. Experiments demonstrate notable performance improvements across various benchmarks and model types.
Why it matters: This method offers a novel and efficient way to boost model performance without the computational cost of traditional fine-tuning.
Researchers reformulate tabular foundation models (TFMs) to address structural challenges in discrete choice estimation, such as choice-set dependence and consumer heterogeneity. Their approach encodes these factors within a row-based learning framework and, when evaluated on a yogurt scanner panel, outperforms hierarchical Bayesian estimation by 8% in holdout log-likelihood and 3.6% in hit rate, while being 16 times faster. The method is particularly effective in medium-data regimes (10–40 purchase occasions per consumer), where traditional Bayesian methods can distort estimates for atypical consumers.
Why it matters: This work demonstrates a significant advance in applying foundation models to consumer choice estimation, offering both improved predictive performance and substantial computational speedups over established methods.
A new preprint demonstrates that predictive coding (PC) can avoid the non-local Jacobian-transpose operation by factoring it into three locally available terms for layers with frozen normalization. The resulting method, WF-Act-PC, removes the need for autograd backward passes in error transport and, on benchmarks like CIFAR-10 and Tiny-ImageNet, matches or exceeds backpropagation performance on deeper architectures, outperforming previous PC methods.
Why it matters: This work addresses a longstanding obstacle to biologically plausible learning by eliminating a key non-local operation in predictive coding, narrowing the performance gap with backpropagation in deep networks.
SteinGate proposes a boundary-aware distributional safety certificate for safe reinforcement learning, leveraging Kernelized Stein Discrepancy to robustly detect rare catastrophic cost events. The method dynamically alternates between reward-seeking and recovery policies based on deviations in the cost distribution's tail, aiming to reduce constraint violations during training. Experimental results on continuous-control benchmarks show that SteinGate lowers both the frequency and severity of safety violations while maintaining competitive performance compared to state-of-the-art methods.
Why it matters: This work offers a novel approach to addressing rare but severe safety failures in reinforcement learning, potentially improving the reliability of RL systems in safety-critical applications.
A new preprint benchmarks Kolmogorov-Arnold Networks (KANs) against Multi-Layer Perceptrons (MLPs) on 12 structured tabular classification tasks. The study finds that KANs achieve statistically significant accuracy improvements over MLPs, particularly in binary and multiclass settings, but require substantially more parameters and computational resources. The authors recommend KANs for high-precision needs and MLPs for efficiency in resource-limited scenarios.
Why it matters: This work provides empirical evidence to inform model selection for structured data, clarifying the trade-off between accuracy and computational efficiency when choosing between KANs and MLPs.
A new preprint demonstrates that temperature scaling, a widely used model calibration technique, systematically misrepresents model reliability when ground-truth labels are soft or distributional, such as those from crowd-sourced human annotations. Evaluating nine model configurations on the CIFAR-10H and ChaosNLI datasets, the study finds that temperature scaling calibrated on hard labels consistently underperforms an oracle calibrated on soft labels, with calibration gaps notably larger in language tasks (mean 0.079) than in vision tasks (mean 0.003). The results hold across model scales and with an alternative calibration method, multiclass isotonic regression.
Why it matters: The findings highlight that standard calibration protocols relying on majority-vote labels can give a misleading sense of model reliability in real-world scenarios with inherent label ambiguity, posing risks for safety-critical AI deployments.
A new framework, Self-Correcting Coupled Markov Jump Processes (SC-CMJP), is introduced to enable concurrent image and text generation by coupling masked diffusion models across modalities. The associated training-free sampler, CO₂Jump, demonstrates state-of-the-art performance on joint multimodal tasks such as image editing and visual reasoning, as shown on newly released large-scale benchmarks.
Why it matters: This work represents a significant advance in multimodal AI by enabling real-time, cross-modal correction and coherent joint outputs, addressing limitations of previous systems that treated modalities separately.
Researchers introduce EXPLORE, a framework that integrates simulator-guided Monte Carlo Tree Search with transformer-based language model decoding to improve analog circuit topology generation. On a 6-component benchmark with tight tolerance, EXPLORE achieves a 65% success rate, outperforming one-shot generation (12%) and sampling-and-filter baselines (33%). The framework also reduces mean squared error by over 20% compared to sampling-and-filter under the same search budget.
Why it matters: This work demonstrates a significant advance in automating analog circuit design by enabling language models to generate complex topologies more reliably through structured search.
Researchers have introduced HEDGEHOG, a six-stage filtration benchmark designed to rigorously evaluate generative molecular models for drug discovery. When 23 different generators were tested on 230,000 molecules, only 0.65% of the generated compounds passed all stages, which include medicinal chemistry, synthesis feasibility, and 3D docking constraints. This finding highlights that current AI models rarely produce molecules that meet all practical requirements for drug candidates.
Why it matters: HEDGEHOG exposes a significant gap between the theoretical capabilities of AI-driven molecular generators and their practical utility in real-world drug discovery.
Agora is a system that enables efficient pretraining of large language models using heterogeneous, individually owned GPUs connected via the internet. By combining bandwidth-efficient pipeline parallelism with fault-tolerant collective operations, Agora successfully trained an 8.6B-parameter model on 500B tokens using 330 contributor nodes over 40 days. The system achieved 63% of the efficiency of a centralized H100 GPU cluster, demonstrating the feasibility of large-scale, decentralized model training.
Why it matters: This work shows that large-scale AI model training can be decentralized and permissionless, potentially broadening access to frontier AI development beyond traditional data centers.
SinAE is a flow-matching autoencoder that uses a single vanilla Transformer architecture to handle molecules, crystals, and proteins, without relying on domain-specific operators. It achieves near-lossless reconstruction across these domains and demonstrates strong generative performance, with joint training on molecules and crystals leading to improvements in both domains.
Why it matters: This work provides a unified approach for generative modeling across diverse atomic systems, enabling cross-domain transfer and potentially alleviating data scarcity issues.
Researchers present SLEUTH, a structured epistemic working memory system that enables language agents to explicitly track confirmed facts, active hypotheses, and open questions during multi-hop reasoning tasks. Evaluated on five multi-hop benchmarks, SLEUTH improves accuracy by up to 11 points on 4-hop chains and outperforms Reflexion without requiring multiple episodes. The study demonstrates that maintaining an organized epistemic state is crucial for scaling multi-hop reasoning, independent of raw model capability.
Why it matters: This work addresses the core challenge of context dilution in language agents and shows that structured reasoning organization, rather than model size alone, is key to reliable multi-step tool use.
A new preprint investigates the premise that quantization loss in large language models can be effectively predicted from per-layer sensitivities. The authors show that per-layer effects account for 85–93% of loss variance at 4-bit precision, and introduce a coverage model and additive predictor that achieve low KL divergence on models ranging from 30B to 355B parameters. Notably, their approach maintains performance on code and reasoning tasks below four bits, outperforming gradient-based methods in these regimes.
Why it matters: This work offers a theoretically grounded and empirically validated approach for mixed-precision quantization, potentially enabling more efficient deployment of large language models with minimal accuracy loss.
A new preprint challenges the prevailing belief that information dependency, such as rote memorization, is responsible for training data exposure in image reconstruction attacks. The authors demonstrate that adversarially non-robust features are the actual cause and introduce Anti Adversarial Training (AT-AT), a method that intentionally learns such features to improve both privacy against model inversion attacks and model accuracy.
Why it matters: This work revises the understanding of training data privacy in machine learning and highlights a new tradeoff between privacy and adversarial robustness.
Researchers present conDitar-dev, a conditional diffusion-based framework for structure-based drug design that generates ligands with strong binding affinities and improved ADMET properties. The model outperforms state-of-the-art baselines on a benchmark of human disease targets, achieving an average binding score of -8.85 kcal/mol and up to 73% improvement in ADMET properties. Experimental validation on PD-L1 and CSF1R targets yielded molecules with micromolar binding affinities and nanomolar IC50 values, demonstrating real-world applicability.
Why it matters: This work advances AI-driven drug discovery by demonstrating a method that generates molecules with both computational and experimental validation, potentially accelerating the development of new therapeutics.
A new study proposes a feature-guided zero-shot framework that leverages large language models (LLMs) for early chronic kidney disease (CKD) screening without the need for dataset-specific training. By selecting a compact set of clinically meaningful and readily available features, the researchers evaluated four LLMs across three heterogeneous CKD datasets from different countries. The models achieved consistent and statistically significant improvements in balanced accuracy and probability estimates, reaching performance levels suitable for screening purposes.
Why it matters: This work shows that LLMs can enable training-free, clinically meaningful CKD screening using minimal, community-accessible features, potentially improving access to screening in resource-limited settings.
Researchers have introduced JoPMol, a generative model that unifies gene expression profiles, molecular structure text, and chemical property data to enable precision molecular design. JoPMol demonstrates superior performance compared to state-of-the-art methods across several evaluation metrics and exhibits strong generalization in transfer tasks and biologically grounded simulations. The model's code is publicly available.
Why it matters: This work represents a notable advance in AI-driven drug discovery by enabling coordinated control over both biological and chemical conditions for personalized molecular design.
A new preprint systematically studies how the training duration of domain expert models affects the quality of merged multi-task models. The authors fine-tuned experts on five domains and three model sizes, evaluating five merging methods at various training checkpoints. They found that simple averaging performs worse as experts overfit, while sparsification-based merging methods achieve their best results well past the validation loss optimum. The study concludes that training duration and merging method should be selected together for optimal results.
Why it matters: This work provides actionable insights for practitioners seeking to improve multi-task model merging, a key technique for combining specialized models without co-training.