TutorMoments: Do AI tutors know when to help and when to hold back?

Allen Institute for AI has a new benchmark called TutorMoments, and it's less about whether chatbots can do math than whether they know when to shut up about it. The setup replays real one-on-one tutoring sessions, pulled from a U.S. high-dosage tutoring program serving mostly Title I students in grades 2 through 7. Experienced math teachers flagged 1,500-plus moments in 462 transcripts where a human tutor had to make a judgment call: ease the problem for a stuck kid, or push them to keep grinding through the confusion. AI2 then hands the transcript to an LLM right at that fork in the road, pairs it with a simulated student, and watches what happens over five turns.

The results are pretty unflattering for the industry's favorite chatbots. Given nothing but a generic instruction to "tutor well," seven major models — Gemini 2.5 Pro, Claude Opus 4.8, GPT-5.5, DeepSeek V4 Pro and others — consistently over-helped. They handed out support even when the moment called for the student to sweat it out, and they rarely pushed for the kind of rigor that research links to actual learning. That tracks with something AI2 points out explicitly: these models are trained to be helpful, and helpful usually means doing the hard part for the user. In a tutoring context, that instinct is closer to a liability than a feature.

Telling the models exactly what trade-off they were navigating helped, but not by much. Every model improved once the prompt spelled out when to scaffold and when to hold back, yet none closed the gap to something resembling reliable judgment, and the spread between models stayed wide. Rigor was the harder skill across the board — models leaned on one trick, mostly asking students to explain their reasoning, while the human tutors in the dataset used a broader mix of moves and were far more willing to simply step back and let a kid work unaided.

Worth noting: human tutors didn't exactly ace this test either, scoring 0.458 on appropriate scaffolding and a rough 0.182 on appropriate rigor, numbers that sit near or below the models' best evaluation-aware scores. AI2 is careful to say this isn't proof AI already tutors better than people — the transcripts were deliberately curated around moments where teaching could have gone better, so the human baseline reflects missed opportunities, not best practice. It's also a benchmark of behavior, not learning outcomes; the simulated "oracle" student can't tell you if a real kid actually understood anything afterward.

AI2 is releasing the de-identified transcripts, the replay code, and the model outputs, framing this as a preview rather than a finished product. The stated goal is nudging AI tutoring companies toward building systems that adapt to a student's actual state rather than defaulting to doing the work for them. Given how much money is currently flowing into AI tutoring products, a public, teacher-grounded yardstick for that specific failure mode is overdue.