Pop Goes the Stack

Round robin isn’t just “not ideal” for LLM inference. According to recent scheduling research, it’s actively harmful, because it treats inference like stateless, interchangeable API traffic when it’s anything but. In this episode of Pop Goes the Stack, Lori MacVittie is joined by F5's Josh Mendoza, Principal Solutions Engineer, to break down why classic load-balancing assumptions fail under LLM workloads, and what to think about instead.

Josh walks through the evolution from early “spray and pray” distribution to smarter approaches that account for server load, workload type, and state. That history matters because AI introduces the same challenge at a new intensity: inference is a heavy compute-and-memory math pipeline, and conversations accumulate state. Once context and KV cache are involved, moving a request to a different server isn’t a clean failover, it’s a forced cache miss and a recomputation penalty that shows up as slower time-to-first-token and higher cost.

They connect this back to patterns teams already understand: VM migration, session persistence, and why “just move it” has always been expensive when the working set is large. LLMs raise the bar because users won’t tolerate latency, and the payload you’d need to move grows as the interaction continues. You also can’t ignore the request itself, since “summarize this” and “write a full analysis” have very different compute profiles, even if they hit the same endpoint.

The main takeaway is simple: don’t panic, but stop treating inference like generic API traffic. Effective LLM scheduling is closer to dispatching the right resources to the right job, with awareness of state, model placement, cache locality, and the true cost of moving work. The tools exist, but the mental model has to change first.