Now in Nature: Retrofitting language models to operate over bytes

Allen Institute for AI says the research behind Bolmo has now been published in Nature, alongside new checkpoints on Hugging Face. The release pushes the same byte-level approach beyond the Olmo models that Bolmo started from, and it also includes Stage 1 checkpoints for researchers who want a quicker place to begin experimenting with the architecture.

The basic idea is simple enough. Most language models chop text into subwords, which usually works well but can stumble on spelling, rare words, unusual strings, or text that doesn’t fit neatly into a fixed vocabulary. Bolmo works lower down the stack, directly on the bytes that make up text — letters, punctuation, symbols, and everything else computers use to represent characters.

That shift buys flexibility. AI2 says byte-level models can better capture fine-grained structure such as whitespace and different writing systems without being tied to a fixed vocabulary. And because bytes are the foundation for digital data more broadly, the same approach could one day be useful beyond text, including for images and audio.

The hard part has always been performance. Training byte-level models from scratch has been expensive, which made them hard to keep up with as subword models kept improving. AI2’s answer is a process it calls byteifying: start with an already capable subword model, then convert it into a byte-level one with a relatively short extra training run.

That method produced Bolmo 1B and Bolmo 7B from AI2’s open Olmo models, and the lab says they were the first fully open byte-level language models competitive with strong subword models across a broad range of tasks. The Nature paper shows the trick works beyond Olmo too. AI2 applied it to Qwen 3 8B and Llama 3 8B, creating Bwen 8B and Blama 8B. Both come close to the models they were derived from, and Bwen 8B is described as the strongest byteified model yet, ahead of Bolmo 7B on AI2’s aggregate evaluation suite.

People are already poking at Bolmo for niche uses, including a Bolmo 7B fine-tune for Russian-to-English poetry and lyric translation. That fits AI2’s broader pitch: publish the model, publish the recipe, let other researchers test the assumptions, and maybe stop pretending one representation of text is the only one that matters.