Datalab Releases OmniParseBench: An Open OCR Benchmark With 16,288 Pass/Fail Tests to Audit OCR Parsers
MarkTechPost Asif Razzaq
Datalab launched OmniParseBench, an open OCR test suite with 16,288 pass/fail checks. It’s meant to show which parser works on documents like yours, not just win a leaderboard.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Datalab has put out OmniParseBench, an open benchmark for document parsers that judges them with yes-or-no tests instead of fuzzy scoring. The idea is pretty simple: give OCR systems real pages, ask specific questions about what’s on them, and see what they actually get right.
The scale is non-trivial. OmniParseBench covers 16,288 tests across 2,937 pages drawn from 2,343 documents and spans 92 languages. Datalab says the dataset and code are public, and the scorer runs offline with no API keys required. You install it with uv pip install omni-parse-bench, and it needs Python 3.11 or newer.
What Datalab is really pushing here is a more practical question than the usual benchmark race: which parser fits this kind of document? That matters because the benchmark is built around pass/fail checks on individual page contents, not a single model judge or a broad, hand-wavy score. Each test is tagged by structure, rendering, role, script and language, so results can be sliced later without changing the verdict.
The headline score is the mean of three families: text, tables and layout. Within those families, tests check things like whether text appears, whether one line comes before another, whether a table cell sits under the right headings, or whether a block is a heading, text or table. Numbers have to match digit for digit, though 1,240 is accepted as 1240, and equations are compared through KaTeX rendering.
On launch, Gemini 3.8 Flash leads with 92.9, and Datalab’s own accurate setting is close behind at 92.7. The top three reported systems sit within 1.3 points of one another, which is a nice reminder that these things are rarely separated by magic. Datalab also flags that two models helped build some tests, and it says the best benchmark is still your own documents. That’s not a slogan; it’s the whole point.
My take — AI-written commentary, not fact-checked reporting
Open benchmarks only matter when they punish everyone, including the company shipping the tool. Datalab at least knows enough to put its own parser in second place and say so out loud, which is rarer than it should be. The bigger lesson is that OCR needs less leaderboard theater and more boring, auditable tests that resemble actual work.
Read more about this at: MarkTechPost