MolmoPoint: Better pointing architecture for vision-language models

Pointing seems like a trivial thing for an AI model to do until you realize how much depends on it. A robot arm needs to know exactly where to grip a mug. A computer-use agent needs to click the right pixel in a screenshot. A video model tracking a person across frames needs to keep pointing at the same spot, frame after frame. Most vision-language models handle this by generating coordinates as text or by predicting tokens tied to coordinate bins, which is a clunky solution — it burns output tokens, forces the model to learn an arbitrary numbering system, and tends to fall apart at high resolution.

AI2's answer, released today as MolmoPoint, throws out the coordinate system entirely. Instead of describing a location in text, the model points by directly selecting parts of its own visual input. It does this in three steps using new special tokens: pick a coarse patch, refine it to a subpatch using finer ViT features, then predict a precise location inside that subpatch. Rotary embeddings track the distance between successive points so the model doesn't double back on itself, and a dedicated stop signal lets it declare 'no more points' instead of being forced to keep guessing. The whole scheme needs just three tokens per point instead of eight.

The release comes in three flavors: MolmoPoint-8B for general image and video work, a GUI-specialized 8B model, and a 4B model tuned for video. AI2 also built MolmoPoint-GUISyn, a synthetic dataset of 36,000 high-resolution screenshots with over 2 million labeled points, generated by having an LLM write HTML mockups of software and then using the Playwright browser automation tool to extract exact bounding boxes for every element. There's also a new tracking dataset, MolmoPoint-TrackData, layering human-annotated and synthetic occlusion-heavy tracks on top of AI2's earlier Molmo2-VideoPoint work.

The numbers back up the redesign. MolmoPoint-8B hits 70.7% on PointBench versus 68.7% for the prior Molmo 2 model, and 89.2 F1 on PixMo-Points versus 85.2. On GUI grounding, the specialized model scores 61.1 on ScreenSpot-Pro and 70.0 on OSWorldG, which AI2 says is state of the art among fully open models — and when they controlled for training data by fine-tuning Molmo 2 on identical GUI examples, the grounding-token approach still won by 2 to 9 points, meaning the gain comes from the architecture, not the data. Video tracking improved by 5.7 J&F on Molmo2-Track, and human raters preferred the new model's video pointing 59% of the time.

What's maybe more interesting than the leaderboard wins is how much easier this version is to train. With just 8,192 examples, MolmoPoint already beats the coordinate-based baseline by roughly 20 F1 points, and it reaches peak performance faster during full pretraining too. That's a strong hint that spelling out coordinates was always an awkward tax the model had to pay, and once you let it point at its own internal representations instead, learning to ground language in images gets a lot more natural.