Honest answer up front: the numbers below are internal benchmarks, run by the project against its own suites with an LLM judge, not independent evaluations. Within that frame they are taken seriously: the judge is pinned to a fixed provider to avoid self-grading inflation, holdout sets exist where the prompt could have been contaminated, and the repo publishes the failure cases alongside the wins.
The headline numbers
| Surface | Suite | Headline | Caveat |
|---|---|---|---|
Fluid blank (capital of france _) | 137 cases, 8 categories | 100% on Cerebras gpt-oss-120b at ~262ms; 99.7% ±0.4 over 5 repeat runs | OpenAI's small nano model scored 40.9% on the same suite: provider choice matters enormously |
| Ambient context | 176 cases (137 standard + 39 ambient) | Shipping bar is 175/176; a prompt change that dropped it to 166 was reverted | One standard-suite case is a known judge flake |
Sentence cues (more-formal) | 5-provider sweep | 100% precision: 150 reject decisions across 5 providers without one false positive; recall 91.3-100% | Cases were authored without iterating on model output, but no holdout suite yet |
Fluid config (settings from _) | In-prompt + holdout, 5 providers | 100% precision across 210 reject cases; holdout recall 90-100% | Holdout exists precisely because the in-prompt set became contaminated |
Transform blank (make this formal _) | ~250 cases, grown over time | Low-to-high 80s percent depending on provider; the strongest recent run is 87.6% (gemma-4-31b on Cerebras) | The hardest surface; long-text and linked-concept rewrites are the weak categories |
How the numbers are kept honest
Three methodological choices in the repo's own benchmark docs are worth knowing. First, the LLM judge is pinned to one provider regardless of which provider is being tested, because self-judging was measured to inflate accuracy by roughly 5 points. Second, the docs state a practical noise floor: treat ±3 points of accuracy and ±15% latency as run-to-run variance on single-trial numbers. Third, where a prompt was tuned against its own test set (fluid config), a separate holdout suite was built, and the honest holdout recall (90-100% rather than a clean 100%) is the number reported.
An interesting internal finding
The benches drive model choice, not just marketing. A June 2026 comparison found Cerebras' gemma-4-31b within a point of the default gpt-oss-120b on fluid-blank quality (98.5% vs 99.3%) while running roughly twice as fast, and a July re-run had it winning transform-blank outright (+4.3 points). The default is still gpt-oss-120b, but the data is in the repo for anyone who wants to switch.
What the benchmarks don't measure
The suites call the LLM source classes directly, so they measure model quality: accuracy, latency, cost per case. They deliberately do not exercise the runtime layer around them (span tracking, merges, key handling), which is covered separately by 500+ unit and scenario tests plus an end-to-end harness. And nothing here is a third-party evaluation: if you want to check the claims, the suites and their experiment logs live under tests/benchmarks/ in the repo and are runnable against your own keys.
Related: How OpenCues stays fast · Cerebras vs Groq · LLM providers · Why use OpenCues? · GitHub repo