Linear representations of grammaticality in neural language models

arXiv cs.CL · 6 hours ago

Researchers investigated whether neural language models encode grammaticality in their internal representations rather than just assigning higher probabilities to grammatical sentences. Using mass-mean probing on pretrained models, they found that grammatical and ungrammatical sentences are systematically separated in representational space independent of other sentence properties like word frequency and plausibility. This encoding generalizes across multiple grammatical phenomena and some languages, suggesting grammaticality forms a coherent representational dimension in current language models.