Liquid AI Open-Sources Pipette: A Reproducible Benchmarking Suite That Measures On-Device Models, Quantization, Runtime and Hardware Together
MarkTechPost Asif Razzaq
Liquid AI open-sourced Pipette, a benchmark for models on real devices, not just in servers. It shows the phone, runtime and quantization can change the result more than the model name.
Based on reporting by MarkTechPost, Asif Razzaq — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Liquid AI has released Pipette, an open-source benchmarking platform built with Artificial Analysis to measure foundation models the way they actually get used on edge devices: as a full stack, not a neat little model card.
That matters because the usual server-side, full-precision numbers can be fantasy once a model lands on a phone. Pipette treats the unit of measurement as a deployment configuration — model, quantization, runtime and device together — and the launch dataset already spans more than 1,000 such configurations across 30+ models.
The launch covers five on-device performance metrics, llama.cpp builds for macOS, iOS, Windows and Android, and context lengths from 256 to 8,192 tokens. The first verified results are from a MacBook Pro with M5 Max, an iPhone 17 Pro and a Galaxy S26 Ultra. Liquid AI also says AMD Ryzen AI Max+ 395 and Radeon 8060S results are coming soon.
The point of the benchmark shows up in the numbers. On a Galaxy S26 Ultra at Q4_K_M, Granite-4.0-H-350M keeps 78.4% of its decode throughput as context grows from 256 to 4,096 tokens, while Granite-4.0-350M keeps only 33.8%. Same general size. Very different behavior.
Other comparisons are just as awkward for anyone who likes tidy rankings. On the same phone and at 2,048 input tokens, LFM2.5-8B-A1B decodes 2.4 times faster than Qwen3.5-4B and 2.6 times faster than Ministral-3-3B-Instruct-2512, even though it activates only 1.5B of its 8.5B parameters per token and still peaks at 5.29 GiB of memory because all expert weights sit around in RAM anyway. Speed, memory and quality refuse to line up on command.
Pipette also separates performance from quality. The quality scores come from IFBench, GPQA Diamond and MATH-500, run on NVIDIA H100 80GB reference systems and then matched to on-device runs with the same model and quantization. That keeps the phone from pretending to be a lab bench, which is a nice change of pace.
My take — AI-written commentary, not fact-checked reporting
This is the kind of boring infrastructure the industry desperately needs, which is exactly why it’ll be ignored until a product team ships the wrong model on the wrong phone. The big lie in AI is still that a single benchmark number means anything useful; Pipette at least puts that nonsense under a brighter light. Context, runtime and hardware are not footnotes. They’re the whole bill.
Read more about this at: MarkTechPost
Related stories
Liquid AI Releases LFM2.5-2.6B: An On-Device Agentic Model With 128K Context, Tool Calling, And Open Weights
MarkTechPost · 4 weeks ago ·
44
All of AI benchmarking at your fingertips
IBM Research · 1 month ago ·
1
Together AI delivers fastest inference for the top open-source models
Together AI · 9 months ago ·
34