Falcon-Edge: A series of powerful, universal, fine-tunable 1.58bit language models.

TII's Falcon-Edge project tackles a problem that's quietly annoyed everyone working on BitNet models: you could shrink them, but you couldn't really touch them afterward. Once a model gets quantized down to ternary weights of -1, 0, and 1, it typically becomes a frozen artifact good only for inference. Falcon-Edge breaks that pattern by building fine-tunability into the pre-training process itself, so a single ~1.5 trillion token training run on TII's internal data mixture produces three usable outputs at once: a standard bfloat16 model, a native BitNet checkpoint, and a pre-quantized variant specifically shaped for further tuning.

The technical trick is simpler than it sounds. BitNet's inference math already quantizes activations to int8 and rescales them back, which the team realized creates a decent approximation of the original weights if you inject the weight scale after quantization rather than adjusting weights post-training the usual way. Run that logic end-to-end on the 1B and 3B base models and the bfloat16 versions hold up close to their unquantized cousins on the old Hugging Face leaderboard v2 benchmarks, beating or matching similarly sized models. That's the headline claim: ternary training doesn't have to mean giving up competitive performance, and it doesn't have to mean giving up flexibility either.

To make the fine-tuning story real rather than theoretical, TII shipped onebitllms, a small Python package that plugs into existing tools like Hugging Face's trl library. It converts prequantized checkpoints into a trainable BitNet format, hands them off to standard fine-tuning pipelines, and quantizes the result back down afterward. Right now only full fine-tuning works — no LoRA or other parameter-efficient methods yet — which the team openly flags as an unsolved problem worth chasing, especially once BitNet models grow past the 1B-3B range where full fine-tuning gets expensive fast.```)

What's notable here isn't just the models themselves but TII being upfront about the rough edges. They estimate roughly 20% training overhead versus standard bfloat16 pretraining, they haven't fully closed the performance gap between the ternary checkpoint and its float counterpart, and GPU inference kernels for BitNet still lag behind what's possible on CPU via projects like bitnet.cpp. None of that undercuts the release — it's genuinely useful groundwork — but it reads more like a research checkpoint than a finished product, with multi-modal BitNet models and better GPU kernels explicitly on the wishlist rather than delivered.