Home > FAQs > Rate limits

How does OpenCues handle rate limits?

Rate limits are the most common failure on free LLM tiers, and OpenCues' answer is structural: configure keys for both Cerebras and Groq and the runtime routes around transient failures automatically. Both providers serve the same gpt-oss-120b weights, so a failover changes nothing about quality.

What triggers a failover

The fallback fires on transient failures: HTTP 429 rate limits, 5xx server errors, network blips, and empty response bodies. When one hits, the runtime rewrites the request URL for the peer provider, swaps the auth header to the fallback's key, translates the model name between the two providers' formats, and re-issues the call. The response comes back as if nothing happened; the caller never sees the switch.

What never retries

400-class client errors are never retried, deliberately: they mean the request itself is malformed and would fail identically on the fallback. This matters for debugging. A 429 that self-heals is invisible; a 400 surfaces immediately, which is the correct behaviour because it's your configuration (or a bug worth reporting), not the provider's weather.

Why only Cerebras and Groq pair up

Failover only happens between wire-compatible peers, and today that means Cerebras and Groq: same request shape, same open-weight models, so a request built for one can be mechanically rewritten for the other. Cross-shape pairs (Groq to Gemini, say) would need a different request body, so they aren't auto-paired. The broader auto-route order when picking a provider walks Cerebras, then Groq, then Gemini, then Anthropic, then OpenAI, but the transparent mid-request failover is the Cerebras/Groq pair specifically.

Practical guidance for free tiers

Set both keys. This is the whole trick: opencues set-key cerebras ... and opencues set-key groq .... Both have free tiers; two keys roughly doubles your effective headroom and covers each provider's outages with the other.
Blanks show errors visibly. A data blank that hits a rate limit puts the error where the answer would go (NVDA: HTTP 429), so you know what happened; an LLM lookup that can't resolve leaves the _ in place rather than failing silently.
The free pool has its own behaviour. The no-key opencode-zen pool health-caches dead models for 30 seconds and falls through its model list by accuracy. Bear in mind its honest cost: the preferred free model has a median response around 14 seconds, and inputs may be used for training, which is why it's restricted to the blanks bucket.
Local means no limits. Running via Ollama trades latency for immunity: no rate limits, no quotas, no keys, at your hardware's speed. There's no fallback peer for it, though.

If you're seeing persistent throttling rather than blips, check which bucket is doing the volume: word cues fire far more often than blanks, so pointing cues-llm-provider at the provider with your most headroom is the highest-leverage change.