When a model reads a drug's class from its name—not its knowledge

There's a neat trick buried in drug names: suffixes like -pril or -olol often tell you the drug class before you know anything else about the medicine. Turns out language models can exploit that same trick, and a new study on Olmo 3 shows just how often they do it instead of actually knowing anything about the drug in question.

Kaijie Mo at the University of Texas at Austin led the work with collaborators at Northeastern University and MD Anderson Cancer Center. They picked Olmo 3 specifically because it's fully open, weights, training data, documentation and checkpoints included. That mattered because the goal wasn't just to catch the model guessing, but to trace the guessing back to its source in the training corpus.

The team built a diagnostic on Olmo 3 7B Instruct that separates three things a model could lean on when answering a drug question: the affix (the shared ending), the stem (the part unique to that drug), and genuine drug-specific knowledge. By swapping stems and affixes for made-up nonsense strings and watching how answers changed, they could tell which ingredient the model was actually using. The results were not flattering. Depending on how the test was framed, somewhere between 51 and 59 percent of drugs produced answers barely different from answers to a fake drug name, meaning the model showed little sign of real knowledge. Another 12 to 18 percent looked affix-driven, essentially pattern-matching the ending rather than recalling anything specific.

Using Ai2's infini-gram tool to search the exact training corpus, the researchers found a pretty clean relationship: the rarer a drug was in the training data, the more the model's answer leaned on the affix instead of drug-specific facts. That's not shocking once you think about it. If a model barely saw a drug during training, the name's ending is the only signal it has left to lean on.

Mo and her colleagues are careful to say the affix shortcut isn't inherently bad, since those endings do carry real pharmacological meaning. The problem is when that shortcut quietly substitutes for actual knowledge about the exact medication someone is asking about. And because Olmo 3 is open all the way down to its training data, the team could show not just that the shortcut exists, but exactly where it comes from, something far harder to pull off with a closed model.