TalTech's systems achieved first place in both tracks of the Beyond Transcription Challenge (BeTraC), generating SOAP notes directly from long doctor-patient conversation recordings without intermediate transcription. They adapted Voxtral Mini and Voxtral Small models using LoRA fine-tuning and DAPO reinforcement learning with the Open Medical Concept F1 metric as a reward. Independent evaluation found their submissions had the lowest hallucination rate among all entries.
Why it matters: This result shows that reinforcement learning with a concept-matching metric can improve factual reliability in medical summarization from speech, and that text-based fine-tuning can transfer effectively to speech input.
JOR-Bench is a suite of five Japanese-language benchmarks designed to evaluate large language models (LLMs) on operations research (OR) problems, comprising 1,319 problems across linear, mixed-integer, non-linear, and combinatorial optimization. The benchmarks are Japanese translations of established English datasets, enabling direct cross-lingual comparison. Evaluation of seven LLMs shows that strong multilingual models exhibit nearly identical OR formulation accuracy in Japanese and English, with only a -0.3 percentage point difference, though error analysis uncovers subtle language-specific failure modes.
Why it matters: JOR-Bench provides a rigorous tool for assessing LLMs' mathematical reasoning in Japanese, highlighting both the strengths and nuanced limitations of multilingual models across languages.
Researchers introduce KyrgyzLLM-Bench, a benchmark suite for evaluating large language models (LLMs) in Kyrgyz. The suite includes two natively authored datasets (KyrgyzMMLU and KyrgyzRC) and translated, manually post-edited versions of WinoGrande, HellaSwag, BoolQ, and TruthfulQA. Evaluation of 26 models shows that cross-lingual transfer varies by task, with translation artifacts notably affecting HellaSwag performance. Few-shot prompting improves some open-source models on reading comprehension, but results are inconsistent for proprietary models on translated tasks.
Why it matters: This work provides the first large-scale, natively authored evaluation benchmark for Kyrgyz, enabling more accurate assessment of LLM capabilities and cross-lingual transfer in a low-resource language.
A new preprint argues that achieving zero hallucination in large language models (LLMs) cannot be accomplished by the model alone, but must be enforced at the system level. The authors introduce HALO, a six-layer assurance architecture that includes grounded generation, constrained execution, multi-signal verification, calibrated abstention, total traceability, and continuous oversight. The framework is demonstrated on a regulated claims-extraction task.
Why it matters: This work reframes hallucination as a system-level failure mode and proposes a practical, multi-layered architecture for deploying trustworthy AI in regulated enterprise environments.
Researchers introduce ESCUCHA, the first Spanish speech understanding benchmark designed to evaluate large audio language models across diverse acoustic conditions and reasoning abilities. The benchmark features 1,000 human-curated questions paired with 162.9 hours of real-world audio, encompassing multiple Spanish accents and non-normative speech. Initial evaluations show substantial performance gaps between state-of-the-art models and trained humans.
Why it matters: This benchmark fills a critical gap in evaluating audio language models for Spanish under realistic and challenging conditions, highlighting the need for more robust and capable multilingual speech AI systems.
A new benchmark dataset of 1,516 expert-verified Bangla sentences reveals that large language models (LLMs) exhibit a systematic bias toward the dominant meaning when disambiguating Bangla homographs that serve as both personal names and common nouns. The study demonstrates that contrastive chain-of-thought prompting and distilled cultural explanations can reduce this bias from as high as 100% to under 5% in small models, even turning a previously unsuccessful Bangla-specific model into the strongest performer.
Why it matters: This work demonstrates that language-specific pretraining is insufficient for cultural grounding and offers practical methods to mitigate cultural biases in low-resource language models.
C²KV is a unified framework for non-prefix key-value (KV) cache reuse in large language models that jointly optimizes KV extraction and inference-time concatenation. The method learns a composable and compressed KV cache manifold that is position-agnostic, enabling modular KV representations to be reused without modifying the base model. Experiments show that C²KV can achieve up to 17× inference speedup under long-context scenarios while preserving generation quality.
Why it matters: This work addresses the bottleneck of storing and accessing large KV caches in long-context LLM serving, offering significant speedups without loss of accuracy.
Researchers introduce Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that leverages group-level entropy to adapt advantage signals for reinforcement learning alignment of large language models (LLMs) on heterogeneous tasks. GEPO addresses the challenge of varying exploration needs across diverse task groups and demonstrates consistent outperformance over GRPO and other entropy-controlled methods on 13 benchmarks, including math, physics, code, and instruction following.
Why it matters: GEPO offers a practical advance for aligning LLMs on mixed-task datasets, enabling more balanced and effective training across diverse capabilities.
Researchers have found that hidden-state probes intended to measure output correctness in language models often capture the model's own self-judgement rather than objective correctness. By creating cases where the model's self-judgement and objective correctness disagree, they show that the self-judgement signal transfers reliably across tasks, while the objective-correctness signal does not. This indicates that transferability of probe signals does not guarantee they reflect objective truth.
Why it matters: This challenges the reliability of correctness probes for interpretability and safety, as they may conflate a model's self-assessment with actual correctness.
EvolvingWorld is a framework and benchmark designed for co-evolving characters and worlds in interactive literary simulations. It features a Character Agent and an LLM-based World Model that enable persistent, long-horizon state updates across diverse literary settings. The framework is supported by a dataset derived from 57 books, comprising 138,596 training samples and 222 test snapshots, and introduces a trajectory-level evaluation protocol. Experiments indicate that EvolvingWorld can improve the coherence and persistence of character and world development over extended simulations.
Why it matters: This work represents a notable advance in interactive literary simulation by enabling dynamic, persistent co-evolution of characters and worlds, addressing limitations of prior static or isolated approaches.
A controlled study evaluated the impact of biographical personas in system prompts on code generation by large language models (LLMs). The research found that persona effects varied by model: on Claude Opus, a librarian persona led to frequent in-character disclaimers and a drop in code correctness from 0.92 to 0.67, while GPT-5.5 did not exhibit these behaviors. The findings indicate that personas introduce model-dependent behavioral biases rather than consistent quality improvements.
Why it matters: This work highlights that persona-based prompting can unpredictably affect LLM performance, emphasizing the importance of model-specific prompt engineering.
DeLIVeR is a new framework that frames evidence retrieval for automated fact-checking as a reinforced strategic exploration task. It uses a Planner LLM to decompose complex claims into targeted questions, guiding traversal of structured knowledge graphs for high-precision evidence. On the LIAR, FEVER, and PolitiFact datasets, DeLIVeR achieves peak F1-scores of 83.73, 84.57, and 79.70, representing a 10-15% improvement over the HippoRAG2 baseline.
Why it matters: This work advances automated fact-checking by combining reinforcement learning with knowledge graph traversal, enabling more transparent and auditable multi-hop reasoning for misinformation detection.
A new preprint identifies that model collapse during iterative instruction tuning with synthetic data often appears as a polarization of competence, where strong skills are reinforced but weak skills degrade. The authors introduce KITE, a two-stage framework that uses failure-guided data generation and boundary-aware uncertainty curation to address this issue. Experiments across several datasets and open-source LLMs show that KITE achieves more stable improvements compared to strong synthetic-data baselines.
Why it matters: This work offers a practical approach to mitigating performance degradation in LLMs trained with synthetic data, addressing a growing challenge as model-generated data becomes more common.
A systematic evaluation finds that off-the-shelf large language models (LLMs), specifically GPT-5.2 and Gemini-2.5, can match or exceed the performance of six specialized privacy policy analysis tools across tasks such as contradiction detection, regulatory compliance analysis, and summarization. In manual labeling tasks, LLMs achieved precision up to 91.4% and recall up to 70.9%, demonstrating their ability to perform tool-specific functionalities without additional domain-specific training.
Why it matters: This study suggests that general-purpose LLMs could serve as effective alternatives to specialized privacy analysis tools, potentially reducing costs and simplifying compliance workflows.
Researchers have introduced Pancasila-Dilemmas, a dataset of 1,834 dilemma questions based on Indonesia's Pancasila values, to evaluate the value alignment of large language models (LLMs). Testing 50 LLMs, the study found that all models scored below 0.5 on Probability Match Score and struggled most with dilemmas related to Religion and Unity, indicating a significant gap in their ability to capture Indonesian human values.
Why it matters: This work highlights the importance of culturally-specific benchmarks for evaluating the value alignment of AI systems, especially in non-Western contexts.
Researchers introduce VDAR-Router, a training-free, difficulty-aware framework for routing queries to large language models (LLMs). The system generates explicit difficulty analyses for each query and retrieves historical examples with similar difficulty profiles to estimate which model is most suitable, optimizing for both performance and cost. Experiments on three datasets show that VDAR-Router achieves better cost-performance trade-offs than existing routing methods.
Why it matters: This work presents a novel, training-free approach to LLM routing that explicitly incorporates query difficulty, offering a practical way to reduce deployment costs without sacrificing performance.
A new preprint introduces the Signal-based Model Access Risk Taxonomy (SMART), a framework for classifying attacker access to deployed AI systems based on the specific information signals (such as decisions, confidence scores, or intermediate representations) they can obtain. Unlike traditional white-box/gray-box/black-box distinctions, SMART focuses on the practical deployment context and how different output signals enable distinct evasion attack strategies. The taxonomy aims to help organizations better assess and mitigate risks when deploying or procuring AI systems.
Why it matters: This framework offers a more nuanced and practical approach to evaluating and managing security risks in real-world AI deployments, which is increasingly important as AI systems are integrated into critical sectors.
Microsoft researchers have introduced Adaptive Incident Prioritization (AIP), the ranking algorithm powering Defender Queue Assistant, which continuously prioritizes security incidents for analyst review. AIP, now deployed across tens of thousands of customers, achieves 92.8% Precision@10 in expert-reviewed evaluations and measurably increases analyst engagement metrics. The work also extends the Microsoft GUIDE dataset with what is claimed to be the first public label source for security operations center (SOC) queue prioritization.
Why it matters: This research offers a scalable, adaptive solution to alert fatigue in security operations centers, with demonstrated improvements in incident triage and analyst efficiency.
Researchers have introduced Scientific Feasibility Control (SFC), a graph-structured conformal prediction framework designed to provide statistical guarantees for the validity of scientific reasoning in large language models (LLMs). SFC decomposes reasoning into atomic units and dynamically validates each step, branching to alternative paths when scientific violations are detected. On the PhyX physics reasoning benchmark, SFC achieves 50.1% accuracy, outperforming DeepSeek-R1 (49.8%) and GPT-4 (45.8%), and reduces scientific law violations by 73%.
Why it matters: This framework offers a principled approach to improving the reliability and scientific consistency of LLM outputs, which is important for trustworthy AI in research and education.
A new preprint demonstrates that minimizing Expected Free Energy (EFE) is mathematically equivalent to solving a ρ-POMDP with a fixed exploration weight, removing the need for manual tuning of exploration parameters. The authors prove this equivalence for observe-then-commit and factored observation POMDPs, and show through experiments on several benchmarks that the untuned EFE approach matches or outperforms reward-only planning and avoids over-exploration seen with manually tuned bonuses.
Why it matters: This work offers a principled, tuning-free exploration strategy for partially observable decision-making, with potential impact in domains like fault detection and medical screening.