TutorMoments: Do AI tutors know when to help and when to hold back?

Allen Institute for AI just dropped something called TutorMoments, and it's less a benchmark than a mirror held up to how AI tutors actually behave when a student is stuck. Instead of grading whether a model gives a right answer, the tool grades whether a model makes the right call at a specific moment: should it scaffold the problem down to something easier, or push the student to grind through the harder reasoning themselves? That's the judgment call real tutors make constantly, and it turns out language models are pretty bad at it.

The setup is clever. AI2 pulled 462 de-identified transcripts from a real U.S. tutoring program serving grades 2 through 7, mostly Title I schools, and had 27 experienced math teachers mark over 1,500 moments where a tutor had to decide between easing the load or demanding more thinking. Then a model takes over as the tutor for five turns, playing against a simulated student, and an automated scoring pipeline checks whether it scaffolded when scaffolding was needed, pushed for rigor when rigor was needed, or over-scaffolded by making things too easy.

Seven models went through this gauntlet under two prompting styles. With a bare-bones prompt that just says