Latency-Aware LLM Query Routing for Dynamic Workloads
A new preprint introduces a latency-aware query router for large language models (LLMs) that jointly optimizes for latency, accuracy, and cost. The proposed system uses a lightweight estimator to predict time-to-first-token (TTFT) for each query, enabling more informed routing decisions. Experimental results show that this approach can improve accuracy–cost utility by up to 40% compared to standard load-balancing methods, without increasing latency.
Why it matters: Incorporating latency into LLM query routing addresses a key challenge for real-time applications, enabling more efficient and responsive model serving.
Full story at: arXiv AI/ML ↗