Meet FreeToken: An Edge-Native MoE Serving Engine that Runs 753B GLM-5.2 on a Single Workstation GPU

The open-model race has run into an awkward truth: getting the weights is no longer the hard part. Running them still is. FreeToken, from researchers at UC Berkeley and UT Austin, is aimed squarely at that gap. It treats a personal machine as a single elastic inference system, then shifts computation across GPU, CPU, memory, and the machine’s own bandwidth instead of assuming a datacenter rack behind every prompt.

The pitch is bigger than convenience. The team says it can keep a 35B model interactive on an 8 GB laptop GPU, push 284B on a gaming desktop, and even serve GLM-5.2 at 753B on one workstation card. That matters because the models getting released now — Kimi-K3, GLM-5.2, DeepSeek-V4-Flash — are closing in on proprietary systems, but the cost of serving them still lands on the people who can least absorb it: solo developers, small teams, and anyone paying agent token bills out of pocket.

FreeToken’s core bet is Mixture-of-Experts. DeepSeek-V4-Flash, for example, activates only 6 of 256 routed experts in each of 43 layers, so only 13B of its 284B parameters are used for any given token. The catch is that the unused experts still sit around in host memory, and existing engines waste a lot of time moving them at the wrong moment. The paper says prefill can shove almost the entire expert pool across PCIe, static placement misses changing routes during decode, and consumer CPUs simply do not have the memory bandwidth to mop up the rest.

The engine tries to fix that with three ideas. One is a bandwidth-aware split that decides how much of the miss set should run on GPU cache and how much should be computed on the CPU, based on the machine’s measured bandwidth. Another is semantic-aware caching, which remembers where agent workflows usually break context, such as thinking blocks and tool calls, so only the new suffix needs to be re-prefilled after edits. The last is elastic memory management, which rebuilds the GPU cache under a new VRAM budget without restarting the engine or reloading the model pool.

The claims are not shy. On an RTX 5090, the system is said to deliver 77–83 tok/s on Qwen3.6-35B-A3B and 22–25 tok/s on DeepSeek-V4-Flash, while keeping worst-case TTFT under 44 seconds in the tested matrix. On an 8 GB RTX 4060 laptop, the NVFP4 build serves 35B at 39.3 tok/s. On a single RTX PRO 6000, GLM-5.2 runs at 14.9 tok/s, versus 7.3 for llama.cpp. The code is Apache-2.0, on GitHub, on PyPI as freetoken v0.1.2, and available as a one-click desktop app for Windows and Linux at flashml.ai. In other words: this is not just a paper about a clever scheduler. It’s a very direct attempt to make frontier-ish open models something a person can actually keep running.