Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers
MarkTechPost Asif Razzaq
Marcel Rød released Gigatoken, a Rust-based BPE tokenizer library that processes text at 24.53 GB/s on a 144-core EPYC system, outperforming HuggingFace tokenizers by 989x and OpenAI's tiktoken by 681x. The performance gains come from hand-written SWAR-based pretokenization and pretoken caching rather than algorithmic improvements to the core BPE merge loop. The speedup generalizes across multiple CPU architectures and 23 tokenizer families, enabling significantly faster text encoding for large language model workflows.
Why it matters
Gigatoken is an MIT-licensed Rust BPE tokenizer that encodes text at gigabytes per second. On a 144-core AMD EPYC 9565, it runs GPT-2 tokenization at 24.53 GB/s. That is 989x HuggingFace tokenizers and 681x tiktoken, against baselines already written in multithreaded Rust. The gains come from a hand-written SWAR pretokenizer and pretoken caching, not a faster BPE merge loop. The post Meet Gigatoken: A Rust BPE Tokenizer that Encodes Text at 24.53 GB/s, up to 989x Faster than HuggingFace Tokenizers appeared first on MarkTechPost.
