Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers

Tokenization is the boring middle step everyone ignores while obsessing over model architecture. Marcel Rød, a PhD student at Stanford, decided to actually profile it, and the result is Gigatoken, an MIT-licensed Rust library that makes existing tokenizers look asleep at the wheel.

The numbers are almost hard to believe until you see the methodology. On a 144-core AMD EPYC 9565 chewing through 11.9 GB of OpenWebText with the GPT-2 vocabulary, Gigatoken hits 24.53 GB/s. HuggingFace's tokenizers library manages 24.8 MB/s on the same hardware. Tiktoken, OpenAI's own tool, does 36.0 MB/s. That's 989x and 681x respectively, and the gap holds up on very different silicon too — an Apple M4 Max gets 1,268x over HuggingFace, and even a consumer Ryzen 7 9800X3D pulls 106x. This isn't a benchmark trick tied to one chip or one vocabulary; it works across 23 tokenizer families including Llama, Qwen, DeepSeek, and GLM.

What's interesting is where the speed actually comes from. It's not a smarter BPE merge algorithm — that part is basically solved. It's pretokenization, the step most libraries hand off to a regex engine and never think about again. Gigatoken hand-writes a state machine instead, and the optimization log reads like a masterclass in low-level performance work: a regex baseline crawls at 47 MiB/s, a hand-rolled state machine jumps to 380, adding SIMD gets to 462, and a SWAR-based branchless byte classifier pushes past 830. The last big win — dual-cursor instruction-level parallelism — comes from realizing the real bottleneck was a 25-27 cycle latency chain, not raw throughput, and running two cursors in parallel lets the CPU's out-of-order engine fill in the gaps. That step alone squeezes out another 25% or so, landing at 1,049 MiB/s single-threaded.

The second lever is pretoken caching — if a word has already been tokenized, look it up instead of recomputing it — paired with careful minimization of Python overhead. The optimization log is refreshingly candid about the dead ends too: a hot/cold code split that regressed performance because it blocked LLVM's inlining, a two-pass buffer that was correct but slower due to memory traffic, profile-guided optimization that did nothing because the loop was already branchless. Not every idea works, and Rød says so plainly.

Caveats matter here. The comparisons aren't perfectly apples-to-apples — Gigatoken finds its own document boundaries in whole files, while HuggingFace and tiktoken are tested on pre-split chunks. SentencePiece vocabularies like Gemma and CodeLlama only see 7-10x gains, not four-digit multiples, and WordPiece isn't supported at all. An independent run on KrabArena, using a much smaller 4-vCPU Xeon VM, still found Gigatoken 26x faster than tiktoken and 83x faster than HuggingFace tokenizers, which suggests the core claim survives outside the original author's benchmark rig.