Study: LLM Alignment Adds No Additional Coverage Over Regex Filters for Natural-Language Harmful Requests
A new arXiv preprint finds that, in production-like settings, adding LLM alignment to a regex filter does not increase the block rate for natural-language harmful requests—coverage remains at zero across tested categories. However, when adversarially-framed probes are used, an LLM-based judge can detect refusals that simple substring matching misses, indicating that alignment's effectiveness depends on the evaluation metric. The study uses a locked corpus and multiple probe types to isolate these effects.
Why it matters: The findings question the added value of LLM alignment for safety in scenarios where regex filters are already deployed, potentially impacting how LLM safety pipelines are designed.
Full story at: arXiv AI/ML ↗