EPISODE · Dec 18, 2025 · 40 MIN
vLLM - LLM Serving Optimization: Paging, Routing, and Ranking
from The Gist Talk · host kw
This episode primarily focus on optimizing the efficiency and fairness of serving Large Language Models (LLMs) under high load conditions. One key source introduces PagedAttention and the vLLM serving system, which uses operating system-inspired paging techniques to efficiently manage the dynamic Key-Value (KV) cache memory, drastically reducing memory fragmentation and increasing throughput by 2-4x compared to state-of-the-art baselines. Another source focuses on improving LLM serving by proposing a ranking-based scheduling algorithm that approximates shortest-job-first strategies, leveraging prediction to alleviate Head-Of-Line (HOL) blocking and demonstrating significantly lower latency and higher throughput than First-Come-First-Serve (FCFS) and other methods. Finally, a third source addresses the challenge of ensuring fair LLM access in multi-tenant platforms, identifying the inadequacy of existing fairness approaches due to diverse application characteristics and proposing FairServe, which uses throttling and weighted scheduling to manage abusive user behavior
What this episode covers
This episode primarily focus on optimizing the efficiency and fairness of serving Large Language Models (LLMs) under high load conditions. One key source introduces PagedAttention and the vLLM serving system, which uses operating system-inspired paging techniques to efficiently manage the dynamic Key-Value (KV) cache memory, drastically reducing memory fragmentation and increasing throughput by 2-4x compared to state-of-the-art baselines. Another source focuses on improving LLM serving by proposing a ranking-based scheduling algorithm that approximates shortest-job-first strategies, leveraging prediction to alleviate Head-Of-Line (HOL) blocking and demonstrating significantly lower latency and higher throughput than First-Come-First-Serve (FCFS) and other methods. Finally, a third source addresses the challenge of ensuring fair LLM access in multi-tenant platforms, identifying the inadequacy of existing fairness approaches due to diverse application characteristics and proposing FairServe, which uses throttling and weighted scheduling to manage abusive user behavior
NOW PLAYING
vLLM - LLM Serving Optimization: Paging, Routing, and Ranking
No transcript for this episode yet
Similar Episodes
Mar 26, 2026 ·1m
Mar 19, 2026 ·34m
Feb 18, 2026 ·11m
Feb 11, 2026 ·45m