@fresh-machine-37269 check what caused this error github.com/OpenHands/OpenHands/pull/16982#… of the pr review automation on this repo

🤖 On it! View progress here: oss-agent-canvas.ngrok.dev/conversations/…

@fresh-machine-37269 Ah, ok, thanks. Tell me what LLM Profile was used

✅ Done! The PR reviewer used the

gpt-5.6-sol

profile — model

openai/gpt-5.6-sol

, running on the OpenAI subscription (Codex backend). That's confirmed two ways: • The reviewer automation builds its agent from

GET /api/settings

→

agent_settings.llm

, which is set to

openai/gpt-5.6-sol

(

auth_type: subscription

,

subscription_vendor: openai

, `base_url: https://chatgpt.com/backend-api/codex`). • The server log at

04:38:32

(right before the review conversation was created at

04:39:27

) shows

[Profile Store] Loaded profile gpt-5.6-sol

. The

usage_limit_reached

/

plan_type: team

error in the logs is the OpenAI subscription rate limit, which matches this profile (not the

deepseek-v4

profile that's the agent-canvas UI default).

@fresh-machine-37269 look at the openhands/agent-sdk repo code. there is a behavior with an LLM fallback. check it out; would it be possible to configure it so that, upon reaching rate limit (this error), it would fallback to another LLMProfile, say GLM-5.x ? (which exist btw in profile store?) WDYT? Just tell me, don't do it.

✅ Done! Yes — this is a real, built-in feature in

OpenHands/software-agent-sdk

, and it's designed for exactly this case. Here's what I found and my take. The mechanism

openhands-sdk/openhands/sdk/llm/fallback_strategy.py

defines

FallbackStrategy

. It triggers fallback on a fixed set of transient exceptions, and

RateLimitError

is explicitly in that list:

_LLM_FALLBACK_EXCEPTIONS = (
    APIConnectionError,
    RateLimitError,
    ServiceUnavailableError,
    LiteLLMTimeout,
    InternalServerError,
    LLMNoResponseError,
)

The incident error (

litellm.RateLimitError: usage_limit_reached

) maps directly to that trigger. On an

LLM

, you set:

llm.fallback_strategy = FallbackStrategy(
    fallback_llms=["glm"],          # profile names in the profile store
    profile_store_dir="...",        # defaults to ~/.openhands/profiles
)

and the SDK will load those named profiles from

LLMProfileStore

, try them in order, and — on success — merge the fallback's cost/token metrics into the primary. Your GLM profile already exists

~/.openhands/profiles/glm.json

is present in the profile store (model

openhands/glm-5.2

, with an encrypted API key), so

fallback_llms=["glm"]

would resolve without creating anything new. (

deepseek-v4

→

openhands/deepseek-v4-pro

is also there.) So: yes, it's possible to configure. The

gpt-5.6-sol

profile/settings LLM just needs a

fallback_strategy

field pointing at

glm

. Caveats / my honest assessment 1. It only fires after retries are exhausted. The active LLM has

num_retries: 5

,

retry_multiplier: 8.0

, min wait 8s / max 64s. In the incident, litellm made 4 attempts before giving up — so you'd see a meaningful delay (potentially a couple minutes) before the fallback even kicks in. If you want near-instant cutover on

usage_limit_reached

, the fallback path as-written still pays the retry backoff first. 2. It's per-call, not "sticky." Each new request restarts from the primary (

gpt-5.6-sol

), so under a sustained quota outage the agent would keep hammering the primary + backoff + fallback on every call — slow and a bit noisy, though it would work. 3. Where it has to live matters for this automation. The PR reviewer builds its agent from

GET /api/settings → agent_settings.llm

and passes that dict straight into

POST /api/conversations

. So the

fallback_strategy

needs to be on the active/settings LLM (i.e. in

gpt-5.6-sol

/ the activated profile), not just sitting in a side profile. 4. Credential/auth asymmetry. Primary is a Codex subscription (

auth_type: subscription

), fallback

glm

is

api_key

auth. Functionally fine, but the fallback model will behave slightly differently (tool-calling fidelity, output style, reasoning), and the subscription path goes through

create_subscription_llm_from_config

, which I'd want to confirm preserves

fallback_strategy

end-to-end before trusting it in prod. 5. No nested fallback chains — the fallback LLM has its own

fallback_strategy

disabled during the attempt, so you can't chain

gpt-5.6-sol → glm → deepseek

in a single call; it's a single level per request. Bottom line: technically yes and low-effort — add a

fallback_strategy: {"fallback_llms": ["glm"]}

to the active LLM profile. It'll catch exactly this

RateLimitError

. The main UX cost is the retry backoff before failover, and you'd be trading a code-quality regression risk (subscription → api-key model difference) for availability. Worth doing, ideally paired with a lower

num_retries

(or a shorter backoff) on the primary so the cutover happens promptly rather than after ~2 minutes of retrying.

@fresh-machine-37269 open a pr to the sdk proposing to do the fallback immediately in the case

usage_limit_reached

or what this case exactly is. then add the fallback profile in the profiles/

⏳ Still working — this is taking longer than this run allows. Follow along at the link above; mention me again for an update.