One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Context
A new method for multi-task on-policy self-distillation is proposed, where each teacher model differs from the student only by a learnable soft prompt, keeping the student's core representations unchanged. This approach allows each task to have its own soft-prompt teacher, enabling a single student model to learn from multiple tasks in parallel. Experiments on Qwen3-1.7B-Base and Phi-4-mini-instruct across four tasks show that this method matches or outperforms full fine-tuning while requiring far fewer trainable parameters. The multi-task variant also maintains general capabilities better than sequential fine-tuning, which often leads to performance degradation.
Why it matters: This approach offers a more efficient and scalable way to train large language models on multiple tasks without sacrificing general performance or requiring extensive parameter updates.
Full story at: arXiv Machine Learning ↗