How Molmo 2’s open release helped an MIT researcher rethink fish tracking
Allen Institute (AI2)
AI2 open-sourced Molmo 2, and an MIT researcher used it to build a fish-tracking model. The surprise: he can now ask it, in plain language, to fix tracking errors.
Based on reporting by Allen Institute (AI2) — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
When AI2 released Molmo 2, it didn’t just publish a model. It opened up the code and training data too, giving researchers a way to inspect how the system worked and reuse its strengths on their own problems. That openness ended up mattering far beyond generic image and video tasks.
Kai Van Brunt, an MIT ’25 graduate working in Sara Beery’s BeeryLab, studies computer vision for ecology. His focus is salmon counts from underwater sonar video, the kind of work that sounds simple until you remember someone has to follow individual fish through messy footage, frame by frame. The counts matter because they help researchers track salmon populations and support conservation and fisheries management.
That’s where the pain starts. Tracking systems can lose a fish, confuse one fish for another, or drift from one animal to the next. Fixing those mistakes by hand can mean correcting predictions across long stretches of video. Van Brunt’s idea was to let a researcher describe the error in ordinary language and have the model repair it automatically. The result was Molmo2Fish, a Molmo 2 adaptation for salmon sonar footage.
Molmo 2 was a good fit because it already handled language and video together. Van Brunt says it was the first model he used that made multi-object tracking feel like something a model could learn directly, without a separate detection step. After fine-tuning on salmon videos, Molmo2Fish performed about as well as a specialized fish-tracking system before any corrections, although results varied across rivers.
The correction step is where the open release really paid off. Van Brunt could inspect the model, understand what it already knew, and prepare his own data accordingly. He found that Molmo2Fish worked especially well when it was given predictions from a conventional fish-tracking system, since the two systems tended to make different mistakes. It still has limits: it can fix some clear errors, but struggles more with its own strongest predictions. It’s a research project, not a deployed tool, but it points to a practical use for open models that closed systems rarely make possible.
My take — AI-written commentary, not fact-checked reporting
This is the boringly important case for open models: not bragging rights, but repairability. A closed system can be impressive; an open one can be adapted when the real world refuses to behave. That’s the difference between a demo and a tool scientists can actually bend to fit the mess they work in.
Read more about this at: Allen Institute (AI2)