Features as Rewards

Hallucinations remain one of the more stubborn problems in AI, mostly because checking whether a claim is true is slow, expensive, and doesn't scale the way math or code verification does. A new technique called RLFR — Reinforcement Learning from Feature Rewards — tries to route around that bottleneck by looking inside the model itself rather than outsourcing fact-checking to a human or another LLM doing web searches.

The method trains lightweight probes on a model's internal activations to spot factual claims, flag likely hallucinations, and grade whether a proposed correction or retraction actually fixes the problem. Those probe scores then become the reward signal for reinforcement learning. Because the probes run on a frozen copy of the model rather than the one being trained, the student can't learn to game the monitor — it has to actually change its outputs, not just its internal representations, to score well.

Applied to Gemma-3-12B-IT, RLFR cut hallucinations by 58% on a held-out set of roughly 20,000 knowledge-heavy prompts, at about 90 times lower cost per check than using Gemini 2.5 Pro with web search as a judge. That cost gap isn't trivial: the team ran roughly 360 optimizer steps for about $2,500, whereas the same budget using an LLM-as-judge setup would have covered maybe 3 to 5 steps. The gains break down in an interesting way — only 10% came from the model itself becoming more cautious after training, while 35% came from inline corrections preventing errors from compounding in later text, and another 12.5% came from those corrections actually being accurate.

What's notable is that the probes kept working even after extensive training, which let researchers use them again at test time for further scaling gains via best-of-32 sampling. That's counter to a common fear with this kind of setup — that a model would just learn to disguise its internal representations rather than fix its behavior. Here, changing behavior turned out to be the easier path, and the probes trained on the base model transferred cleanly to the fine-tuned version, meaning you don't need to run two models at inference time.

The bigger claim buried in this work is about method, not just the hallucination numbers. The researchers frame RLFR as a first proof of concept for what they call 'intentional design' — using interpretability tools as a steering signal during training rather than just as a post-hoc diagnostic. If probes can supervise open-ended tasks like factuality, where clean verification is basically impossible at scale, the same approach might eventually extend to other messy behaviors that current RL methods struggle to reward properly.