FlashServe: Adaptive Kernel Provision for Quantized LLM Serving

Quantization is widely adopted to reduce the memory footprint and computational overhead of large language models (LLMs). Existing LLM serving systems typically rely on a static quantized inference kernel, overlooking the fact that real-world workloads exhibit highly dynamic request rates and batch sizes. Our empirical analysis reveals two important findings in quantized LLM serving. First, the performance of different quantized inference kernels exhibits distinct sensitivity to batch size. Second, motivated by the potential of adaptive quantized inference kernel switching to exploit this performance diversity, switching between quantized inference kernels incurs a non-negligible overhead. These findings give rise to two key challenges: optimal kernel selection under dynamic workloads is difficult, and quantized inference kernel switching is expensive. To address these challenges, we propose FlashServe, a serving system for adaptive quantized inference kernel provisioning. First, FlashServe leverages a latency cost model of both inference and switching latency to guide the optimal quantized inference kernel selection. Second, FlashServe supports finegrained asynchronous kernel switching, which overlaps kernel execution with kernel switching. Finally, FlashServe proactively prepares kernel switching, allowing each kernel switching to be triggered immediately once its corresponding kernel execution completes. We implement a prototype of FlashServe and evaluate it on representative LLM workloads. Experimental results show that FlashServe reduces inference latency by 9.0% - 13.6% compared to static quantized baselines.