Find Before You Fine-Tune: Diagnostic Study of Small LLMs for Cybersecurity QA
Researchers introduce FiT, a diagnostic framework that evaluates small language models (LLMs) on vocabulary, knowledge, and contextualization for cybersecurity question answering. Their empirical study finds that fine-tuning often degrades vocabulary and parametric knowledge in small LLMs, with instruction-focused tuning causing significant knowledge collapse. Pre-fine-tuning FiT scores can predict the direction of post-tuning changes, enabling screening of unsuitable models before adaptation.
Why it matters: FiT could help reduce unnecessary fine-tuning and support safer deployment of small LLMs in critical domains like cybersecurity by identifying unsuitable models in advance.
Full story at: arXiv Computation and Language ↗