Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
A new study analyzes how diffusion language models (DLMs) implement induction, a key mechanism for in-context learning. The researchers find that DLMs develop a bidirectional induction circuit, enabling them to copy information whether the relevant context appears before or after the masked token. The work also provides causal evidence that DLMs compute the global fraction of masked tokens as an implicit timestep, despite lacking explicit timestep embeddings.
Why it matters: This research advances understanding of how diffusion language models process context, highlighting fundamental differences from autoregressive models and informing future improvements in generative architectures.
Full story at: arXiv Computation and Language ↗