PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
PolyQ is a CPU-oriented compiler and quantization co-design that enables fractional-bit large language model (LLM) inference on edge CPUs by assigning per-channel bit-widths and applying compile-time optimizations. The framework achieves 2.4–32.1% perplexity improvement over prior methods at a 3-bit target and reduces activation reorder traffic by up to 70.8%. End-to-end measurements on various CPUs show that PolyQ enables practical, predictable, and energy-efficient low-bit LLM inference across diverse edge devices.
Why it matters: This work demonstrates that fine-grained, low-bit LLM inference is feasible and efficient on widely available CPUs, broadening the accessibility of on-device AI.
Full story at: arXiv Machine Learning ↗