For inline AI the honest answer starts with a reframe: latency matters more than parameter count. A suggestion that lands after you've moved on is noise, so the best model is the most capable one that still answers inside the surface's latency budget. OpenCues splits its routing into per-surface buckets precisely because those budgets differ.
Match the model to the bucket
| Bucket | Surfaces | What it values |
|---|---|---|
cues | Word cues, sentence cues | Keystroke-rate latency; word cues budget ~500ms. Only Cerebras sustained medium reasoning inside it in testing. |
blanks | _ lookups and transforms | Balance; the one bucket where free-pool providers are allowed, since typing _ is explicit consent. |
Each bucket has its own provider/model pair in OPENCUES.md (cues-llm-provider:, blanks-llm-provider:), so a split like fast cloud for cues and local Ollama for blanks is two lines of config.
What the benchmarks actually found
From the repo's own suites (internal benchmarks, LLM-judged, pinned judge): Cerebras gpt-oss-120b scored 100% on the 137-case fluid-blank suite at ~262ms; Groq's copy of the same model 99.3% at 686ms with the best cost-per-correct; Gemini flash-lite and Claude Haiku both held high-80s accuracy on the harder transform-blank suite. The cautionary result: OpenAI's small nano model collapsed on both suites (40.9% fluid-blank, and roughly 21% on the multi-pass transform pipeline), which is why OpenCues' OpenAI default is gpt-5.4-mini, not nano. Small-and-cheap is not the same as fast-and-capable.
The candidates worth naming
gpt-oss-120b (default): best benched accuracy-latency combination, open weights, served by two interchangeable providers. gemma-4-31b: in June-July 2026 benches it came within a point of gpt-oss-120b on fluid-blank (98.5% vs 99.3%) and edged it on transform-blank (+4.3 points) at roughly 2x the speed, but it is a Cerebras private-preview model with a measured multilingual weakness (77% vs 90% on the 30-case multilingual set), so the default hasn't moved while it awaits general availability. Local via Ollama (gemma4:e2b): total privacy, zero per-token cost, but a different regime: ~2-3s per transform and 59.8% on the transform suite, best reserved for the deliberate _ surface.
The selection heuristic
Start with the default and change it only on evidence: if cues feel slow, the cues bucket is the one to tune; if rewrites disappoint, try a stronger model, where seconds of latency are acceptable; if privacy dominates, route prose-bearing buckets through Ollama. And note what OpenCues refuses on your behalf: providers whose terms allow training on inputs are structurally rejected for the prose-bearing buckets, whatever their benchmark scores.
Related: What is gpt-oss-120b? · How to switch models · Ollama vs cloud · GitHub repo