Solving virtual machine puzzles: How AI is optimizing cloud computing

Picture Tetris, except the blocks keep changing shape mid-fall and you have no idea how long any of them will stick around. That's roughly what Google's data centers deal with every second, trying to cram virtual machines onto physical servers without knowing whether a given VM will vanish in five minutes or grind on for five days. Get the packing wrong and you end up with what engineers call resource stranding: scraps of CPU and memory too fragmented to be useful, sitting idle while new jobs queue up.

Google's answer, detailed in new research, is a trio of algorithms named NILAS, LAVA, and LARS, all built around a trick called continuous reprediction. Instead of guessing a VM's lifespan once at creation and living with that guess forever, the system keeps asking itself the same question over and over: given this VM has already run for five days, how much longer does it likely have? The model learns that 88% of VMs die within an hour but account for only 2% of total resource use, meaning the long tail of long-lived VMs is what actually determines whether a cluster runs efficiently or not.

Each algorithm attacks the problem differently. NILAS, already live in production since early 2024, nudges new VMs toward hosts where everything already running is expected to expire around the same time, aiming to create clean, fully empty machines rather than half-full ones. LAVA does almost the opposite on purpose: it deliberately drops short-lived VMs onto hosts anchored by a long-lived one, using them to fill gaps without extending how long that host stays occupied. LARS handles the messier job of rescheduling during maintenance, migrating the VMs expected to live longest first so the short-lived ones simply die off on their own instead of getting shuffled around for no reason.

The harder problem, arguably, wasn't the algorithms — it was making them fast enough to matter. Running a machine learning model on separate inference servers would have created a scheduler that depends on infrastructure the scheduler itself manages, a circular dependency nobody wants in a system this critical. Google's fix was blunt: compile the model straight into the Borg scheduler binary. That cut median prediction latency to 9 microseconds, 780 times faster than the server-based approach, and let the team cache lifetime scores so predictions only refresh when something actually changes on a host.

The payoff, per Google's own numbers, is a 2.3 to 9.2 percentage point jump in empty hosts across pilot and fleet-wide rollouts, plus roughly 3% less CPU stranding and 2% less memory stranding in early tests. Simulations suggest LAVA could add another 0.4 points on top of that, and LARS could cut maintenance-related VM migrations by about 4.5%. None of these numbers sound dramatic in isolation, but at the scale of a company running millions of servers, a single percentage point of reclaimed capacity is real money and real megawatts.