Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks

MarkTechPost · 1 week ago · 6

Datalab released OmniExtractBench, an open benchmark that scores structured extraction from PDFs by comparing predicted JSON fields to gold using an auditable deterministic scorer. The benchmark pools 620 documents from 4 existing extraction suites and packages a PyPI scorer (omni-extract-bench v0.1.7) under Apache 2.0. Vendors’ extraction leaderboards become more comparable and easier to audit because scoring is transparent, content-based for table row alignment, and rules drop null/empty-field padding that can otherwise inflate results.