Tracing distinctive language in AI-written text

Remember GrantaGate, when readers caught a prize-winning short story looking suspiciously machine-written? That mess kicked off a much bigger question that's been nagging at people who study AI text: when a language model spits out a sentence, is it actually generating something new, or just remixing phrases it memorized during training? An AI detector can tell you a passage is probably synthetic, but it can't tell you which words came from where. That gap is exactly what Ai2's infini-gram engine was built to close.

Infini-gram works by indexing enormous text datasets and letting anyone search for how often a specific phrase shows up across them. Tuhin Chakrabarty, a computer science professor at Stony Brook, has been putting it to work on a messier real-world problem: the flood of AI-generated books quietly climbing Amazon's charts. His team's research, recently covered by both The Atlantic and The New York Times, dug into the GrantaGate story itself and found that its odd little phrase "sour tang of fermenting" echoed language from a fan-fiction site. Not proof of plagiarism exactly, but the kind of circumstantial clue that turns a fuzzy AI-detector score into something more concrete.

The numbers get more interesting when you zoom out to Amazon's self-published bestseller lists. Chakrabarty's group compared the 200 highest-earning books flagged as substantially AI-written against 200 with no detected AI text, then checked both against Google Books for rare, distinctive expressions — the kind of phrasing that shows up in five books or fewer and never appears on the open web. AI-heavy books scored 4.4 percentage points higher on this measure. Compare those same AI-heavy bestsellers to actual award-winning literature, though, and the gap exploded to 22.5 points. In a follow-up study, rare-expression coverage hit 43.2% in the top 100 AI-flagged books versus 37.6% in books with no detected AI, and award-nominated literature sat way down at just 19.1%.

What that suggests, per Chakrabarty, is that leaning heavily on borrowed, distinctive phrasing isn't some natural byproduct of good writing — it's something closer to a tell. Post-training usually stops models from copying entire passages verbatim, but shorter chunks of language absorbed during pretraining apparently still leak through often enough to detect at scale. None of this pins down definitively where any single phrase originated, and the researchers are careful to say so. But when dozens of rare, book-specific expressions cluster inside a passage a detector already flagged, that's not nothing.

Chakrabarty's next target is copyright, and it's the part of this work with real teeth. His team wants to use infini-gram to trace whether a model's output reflects text pulled from pirated book copies or leaked manuscripts floating around online, rather than legitimately licensed sources. That kind of tracing only works because Ai2 built infini-gram and its training corpus, Dolma, to be fully open and searchable. Try running this same analysis against a closed model like GPT-4 or Gemini, and you'd hit a wall immediately — there's no dataset to search against. Which is sort of the whole point.