A suggestion that arrives after you've moved on is noise, so OpenCues is engineered around latency budgets at every layer.
- Debounced analysis. Normal typing batches into a 500ms-debounced pass, so the system asks once per pause, not once per keystroke.
- The
_jumps the queue. An explicit blank at the end of your text bypasses the debounce entirely: you asked, so it fires immediately. - Stale answers are discarded. Every in-flight LLM call is generation-tagged; if you've typed since, the late response is dropped instead of overwriting newer text.
- Local first. Static tip cues resolve from memory near-instantly, with no network. Only unclaimed words go to the model, in one batched call per source.
- Warm providers. The default provider (Cerebras) prefix-caches OpenCues' large static prompts at a measured 99.5% hit rate: warm calls land in ~445ms that would otherwise take 800-1200ms cold. Word cues typically return in 200-500ms.
The result is a system that can afford to think on every pause without ever making you wait for it: everything arrives asynchronously into the dim layer, and your typing is never blocked.
Related: LLM providers · What is a cue? · How OpenCues keeps your place · Max thinking · GitHub repo