Vast uses tiered storage to ease AI agent memory demands
SiliconANGLE 1 week ago 40
Vast said it is using tiered storage to reduce AI agent memory pressure by offloading KV caches from GPUs during long-running sessions. It noted that a half a million token session can use one-tenth to one-twentieth of a GPU’s memory. This shifts inference from being limited by GPU memory to a distributed setup using GPU, CPU, and persistent storage, enabling scheduling across a machine fleet.