The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models
Apple Machine Learning Research
The study measures how well language models preserve tree-structured expressions when they are serialized into word problems and then extracted back. It evaluates all pairwise combinations of sixteen models. The result is an empirical test of the “communication bottleneck,” showing how much of that structured content survives the round trip.
Why it matters
When language models reason in chain-of-thought or exchange free-text intermediates, they serialize structured information into natural language. How much tree-structured compositional content survives this bottleneck? We propose a round-trip protocol that answers this question empirically for tree-structured expressions. A generator converts a procedurally generated arithmetic expression into a word problem, a separate extractor recovers the expression from the word problem alone, and symbolic equivalence provides an exact oracle. Evaluating all pairwise combinations of sixteen models yields…