arXiv's eess.AS (Audio and Speech Processing) category covers research on AI for speech recognition, synthesis, and other audio understanding and generation tasks.
A new arXiv preprint finds that combining information from multiple anonymized speech utterances—using audio, prosody, and text—significantly improves the ability to verify a speaker's identity, even when anonymization techniques are applied. The study shows that multimodal systems outperform unimodal ones, and that aggregating as few as five anonymized utterances can reduce error rates by over 15% compared to audio-only methods. This suggests that current anonymization methods may not fully protect speaker privacy when adversaries have access to multiple utterances and multimodal data.
Why it matters: The findings highlight a significant privacy risk, indicating that widely used speaker anonymization techniques may be vulnerable to re-identification attacks using multimodal aggregation.
Research→Official→arXiv Audio and Speech Processing
DCASE 2026 Task 5 introduces Audio-Dependent Question Answering (ADQA), a new benchmark designed to evaluate whether large audio-language models answer questions based on audio content rather than relying on textual priors. The ADQA-Bench evaluation set contains 3,000 items across music, speech, and environmental audio, filtered to remove questions solvable from text alone. In its inaugural run, the top system achieved 58.33% accuracy, with evaluation accuracy dropping by an average of 11.91 percentage points compared to development scores.
Why it matters: This benchmark provides a more rigorous and targeted evaluation of audio-language models by ensuring that performance reflects genuine audio understanding rather than exploitation of textual shortcuts.
Research→Official→arXiv Audio and Speech Processing
GigaSpeechBench is a new benchmark for automatic speech recognition (ASR) that includes 680 hours of human-annotated speech spanning low-resource languages, dialects, accents, domain-specific terminology, and age variation. The benchmark covers over a dozen Middle Eastern and Southeast Asian languages, multiple Chinese dialects, English accents, and specialized domains, with human-annotated translations for 11 languages. Evaluations of leading ASR models and commercial APIs show significant performance drops in these challenging real-world scenarios, revealing major blind spots in current ASR evaluation.
Why it matters: This benchmark exposes critical gaps in ASR robustness for over a billion underrepresented speakers, encouraging the development of more inclusive and realistic evaluation standards.
Research→Official→arXiv Audio and Speech Processing
FlowSonic is a zero-shot music editing framework that leverages a pretrained diffusion transformer with rectified flow to enable text-guided editing of real-world music recordings. The method uses deterministic inversion and cross-attention reuse to preserve musical structure, while a high-order ODE solver enhances numerical stability during editing. Experiments indicate that FlowSonic outperforms existing approaches in tasks such as timbre transfer and genre modification, with improvements in semantic alignment, structural consistency, and perceptual audio quality.
Why it matters: This work demonstrates a significant advance in controllable music editing without task-specific training, potentially impacting music production and audio post-processing workflows.
Research→Official→arXiv Audio and Speech Processing
WhisperVC is a three-stage framework designed to convert whispered speech to normal speech using limited paired data. By separating cross-domain alignment from speech generation, the method achieves competitive results, with DNSMOS 3.07 and WavLM speaker similarity of 0.95 on the AISHELL6-Whisper dataset. The approach demonstrates potential for privacy-preserving communication and as an assistive tool for individuals with vocal-fold impairments.
Why it matters: This work advances whisper-to-normal speech conversion in low-resource settings, enabling practical applications in privacy and assistive technology.
Research→Official→arXiv Audio and Speech Processing
UniPASE is an extension of the PASE framework designed for universal speech enhancement across multiple sampling rates. It introduces the DeWavLM-Omni module, fine-tuned from WavLM via knowledge distillation, to convert degraded waveforms into clean, linguistically faithful phonetic representations. The system reconstructs high-fidelity 16 kHz waveforms and upscales them to 48 kHz, achieving robust enhancement with minimal linguistic hallucination. UniPASE demonstrated superior or competitive performance compared to state-of-the-art models and won first place in the objective evaluation of the URGENT 2026 Challenge.
Why it matters: UniPASE advances the field of speech enhancement by addressing both fidelity and hallucination issues across diverse sampling rates, setting a new benchmark in universal speech enhancement.
Research→Official→arXiv Audio and Speech Processing
RealDESED is a new benchmark dataset for domestic sound event detection, featuring 5,710 real-world audio recordings from 652 participants' homes. The dataset includes 15 sound classes, multi-annotator labels, and extensive metadata, aiming to better reflect the variability and complexity of real domestic environments compared to simulated datasets. A transformer-based baseline achieves a macro-averaged PSDS1 score of 0.731 on the test set.
Why it matters: RealDESED offers a realistic and diverse dataset that can drive the development of more robust sound event detection systems for real-world home environments.
Research→Official→arXiv Audio and Speech Processing
SALMONN-2 is an audio large language model (ALLM) built on a unified self-supervised learning (SSL) encoder, enhanced by a multi-layer feature fusion adapter that aggregates hierarchical representations. The model achieves state-of-the-art performance on several ALLM understanding benchmarks among comparable-scale open-weight models. The study also demonstrates that multimodal in-context learning (MICL) can be effectively acquired through targeted contextual biasing training, rather than emerging naturally.
Why it matters: This work demonstrates that general-purpose self-supervised audio encoders can match or surpass specialized supervised encoders, potentially streamlining the development of versatile audio AI systems.
Research→Official→arXiv Audio and Speech Processing
Researchers introduce NABEATs, a noise-aware audio self-supervised learning model that leverages a reference noise input to help suppress undesired noise in audio representations. NABEATs is trained to estimate clean audio representations from noisy signals, leading to improved performance on downstream tasks in noisy environments and better generalization to previously unseen noise types.
Why it matters: This work advances the robustness of audio AI systems in real-world noisy conditions, which is important for applications such as speech recognition and sound event detection.
Research→Official→arXiv Audio and Speech Processing
ChipChat introduces a novel low-latency cascaded system for real-time on-device voice agents, integrating streaming speech recognition, large language models, text-to-speech, vocoder, and speaker modeling. Implemented in MLX, the system achieves sub-second response latency on a Mac Studio without dedicated GPUs, enabling privacy-preserving, fully on-device processing. The work demonstrates that architectural innovations and streaming optimizations can overcome traditional latency bottlenecks in cascaded systems.
Why it matters: This research provides a practical solution for real-time, privacy-preserving voice-based AI agents on consumer hardware, addressing a key challenge in deploying conversational AI locally.
Research→Official→arXiv Audio and Speech Processing
A preprint introduces Precision-Varying Prediction (PVP), a method that improves the adversarial robustness of automatic speech recognition (ASR) models by randomly varying the inference precision. The approach also enables adversarial example detection using a Gaussian classifier on outputs from different precision levels. When combined with uncertainty-based defenses, PVP increases the difficulty for adaptive attackers, requiring them to introduce more perceptible noise to evade detection. Experimental results show significant improvements in robustness and detection across multiple ASR models, languages, and attack types.
Why it matters: This work proposes a practical and effective defense against adversarial attacks on ASR systems, which are increasingly used in real-world automated applications.
Research→Official→arXiv Audio and Speech Processing
Researchers present X-Translator, a modular, low-cost, open-source speech-to-speech translation system that integrates streaming automatic speech recognition (ASR), machine translation, and prompt-conditioned text-to-speech (TTS). The system is designed for real-time translation in long-form and multi-speaker conversations, addressing challenges such as unstable ASR hypotheses, ambiguous turn boundaries, and speaker consistency. X-Translator is evaluated on translation quality, speech quality, latency, and speaker preservation, with code and a demo publicly available.
Why it matters: X-Translator offers an open, reproducible platform for real-time multilingual speech translation with speaker awareness, helping advance practical deployment in complex conversational scenarios.
Research→Official→arXiv Audio and Speech Processing
Researchers introduce WildElder, a Mandarin elderly speech corpus collected from online videos and annotated with transcription, speaker age, gender, and accent strength. The dataset addresses the scarcity of diverse, real-world elderly speech data for automatic speech recognition and speaker profiling. Experimental results demonstrate the challenges of elderly speech recognition and establish WildElder as a new benchmark for the field.
Why it matters: WildElder provides a much-needed resource for developing and evaluating speech technologies tailored to aging populations.
Research→Official→arXiv Audio and Speech Processing
A study of 1,168 Japanese voice actors demonstrates that AI voice-clone attribution systems based on embedding similarity have a persistent misidentification floor of approximately 2.6%, even when using advanced ensemble methods. Under session-disjoint conditions, this error rate rises to 13%. Generic encoders cause about half of clones from non-enrolled individuals to be wrongly attributed to enrolled actors, while 32% of clones of enrolled targets are missed. Domain-matched encoders reduce, but do not eliminate, these attribution errors, revealing that fixed-threshold attribution remains unreliable and potentially unfair.
Why it matters: This research highlights a fundamental limitation in current voice-clone attribution systems, raising concerns about the protection and fair treatment of professional voice actors.
Research→Official→arXiv Audio and Speech Processing
RobustSpeechFlow is a new training strategy for text-to-speech (TTS) systems that extends contrastive flow matching with length-preserving augmentations to simulate repeat and skip errors. This approach improves alignment robustness and reduces content fidelity errors without requiring external aligners or preference data. On the Seed-TTS-eval benchmark, RobustSpeechFlow reduces word error rate from 1.44 to 1.38 using only 0.06B parameters. On the ZERO500 benchmark, it lowers English character error rate from 0.48% to 0.35% and Korean from 0.81% to 0.57%.
Why it matters: This method offers a practical way to improve TTS content fidelity and robustness, making it easier to integrate into existing pipelines without extra data or tools.
Research→Official→arXiv Audio and Speech Processing
Researchers have introduced Audio-Visual Flamingo (AV-Flamingo), an open-source audio-visual large language model designed for joint understanding and reasoning over long and complex videos. The model employs a three-stage curriculum and a novel temporal reasoning framework, achieving strong results across more than 15 benchmarks. AV-Flamingo outperforms similarly sized open models and is competitive with, and sometimes surpasses, much larger open-weight and closed models, especially on tasks involving long-form audio-visual content.
Why it matters: This work significantly advances open-source multimodal AI by enabling robust joint audio-visual reasoning over long-form videos, a capability previously limited to short clips.
Research→Official→arXiv Audio and Speech Processing
Researchers present Dialogs, a 20.6-hour Russian conversational speech corpus recorded in a professional studio, featuring 3 speakers and 11,796 utterances. The dataset is notable for capturing turn-taking rhythm and expressive prosody, with per-utterance style and emotion labels across 12 categories. Crowd MOS tests indicate that Dialogs achieves higher ratings for expressiveness and conversational naturalness compared to existing studio baselines. A VITS2 model trained on Dialogs demonstrates the corpus's potential for expressive, dialog-like text-to-speech synthesis.
Why it matters: Dialogs provides a high-quality resource for developing more natural and expressive Russian dialog assistants, addressing a gap in available conversational speech data.
Research→Official→arXiv Audio and Speech Processing
A new reconstruction-based method for out-of-distribution (OOD) detection in vocoder recognition has been proposed, utilizing an autoencoder to compress and reconstruct acoustic features from a pre-trained WavLM model. The system assigns each vocoder class a dedicated decoder, and samples that cannot be well reconstructed by any decoder are flagged as OOD. The approach achieves a 10% relative improvement over baseline systems on evaluation datasets, with contrastive learning and an auxiliary classifier further enhancing feature distinctiveness.
Why it matters: This method advances the detection of unknown vocoder-generated deepfakes, addressing a critical challenge in anti-spoofing technology.
Research→Official→arXiv Audio and Speech Processing
WanSong is a pure diffusion-based music generation model that directly produces high-fidelity, multilingual songs up to 5 minutes long, outputting both vocals and background music stems in a single run. The model enables faster inference via step-distillation and supports efficient fine-tuning for downstream editing tasks. WanSong departs from autoregressive and multi-stage pipelines, offering a simpler approach to controllable, long-form music generation.
Why it matters: WanSong introduces a non-autoregressive, diffusion-based method for generating long-form, high-quality songs, representing a notable advance in controllable music generation.
Research→Official→arXiv Audio and Speech Processing
Researchers introduce Echoes, a dataset of 4,468 tracks (131 hours) spanning multiple genres and generated by ten AI music systems, designed to train and benchmark robust deepfake detectors. The dataset enforces semantic alignment between spoofed and bona fide audio to prevent shortcut learning. Cross-dataset evaluations show Echoes is the hardest in-domain dataset and that training on it yields the strongest generalization performance for deepfake detection.
Why it matters: Echoes provides a challenging and diverse benchmark that advances the robustness and generalization of AI-generated music deepfake detectors.