Build Your Own Decision Model

The basic trick here is simple: stop asking a language model to write free-form text and make it choose from a fixed set of answers instead. In the setup described, the model is only allowed to emit A, B, C, D, or E, so it can make a decision in one pass rather than generating token after token the way structured JSON output does.

The example uses Qwen/Qwen3-1.7B and a small Python script that masks out every vocabulary token except the first token for each answer choice. That turns the model into a kind of forced-choice classifier. It still doesn’t guarantee the answer is right, but it does let the model surface a probability for each option, which is tempting to read as confidence.

On a simple test question about the color of the sky, that works exactly as hoped: the model picks B and gives Blue almost all the probability. But on a random sample of CommonsenseQA, the picture is more mixed. The reported accuracy comes in at 725/1221, or 0.5938, with a macro F1 of 0.5844. A quick fine-tune lifts that to 762/1221, or 0.6241, and a macro F1 of 0.6234.

The more interesting part is calibration. When the model is asked a vague question like where you’d most likely find a bat, it snaps to Cave with 0.9978 confidence. That’s the problem in miniature: the number looks precise, but the model is often much less certain than it claims. In the evaluation bins, the 0.9 to 1.0 range is especially noisy: 809 cases sit there, the average confidence is 0.9855, and accuracy is only 0.7009.

Temperature scaling improves that. Using a fitted temperature of 3.797280788421631 flattens the probability curve enough to make confidence line up much better with reality. After that adjustment, the bins track accuracy far more closely, especially in the middle and upper ranges. The author also says the repo includes scripts for building a dataset, evaluating, fine-tuning, and calibrating the model, which is the useful part if you want to try this on bigger models.