<@U0BME18CF7S> we just merged in agent-sdk a quick...
# automations-debug
c
@fresh-machine-37269 we just merged in agent-sdk a quicker fallback LLM use and look at https://github.com/OpenHands/software-agent-sdk/pull/4917#issuecomment-5612767194 too and identify the error that happened there in your environment. look at the We need to configure a fallback LLM for the pr-review automation. can you check automation repo and extensions repo, do they have any pr about fallback? check your pr-review automation, too; and tell me how do we configure e.g. fallback to glm profile?
👀 1
f
🤖 On it! View progress here: oss-agent-canvas.ngrok.dev/conversations/…
✅ Done! Here's what I found across the repos and your environment. The error in the linked comment The comment (
all-hands-bot
at commit
2b97474214e8
) is the pr-review automation reporting
status: error
. That matches issue #4936 ("Hard quota exhaustion delays fallback LLM behind retry backoff"): the production incident returned
usage_limit_reached
for a team-plan model and retried it four times through exponential backoff before the review errored — instead of immediately failing over to a configured fallback. So the root cause is exactly what PR #4917 now fixes (skip retries on deterministic quota errors and go straight to
FallbackStrategy
). Does automation repo have a fallback PR? No LLM-fallback PR. What exists is single-profile selection only: • #120 "Support named LLM profiles in automations" (merged) — the
model
field on the automation +
AUTOMATION_MODEL
env var, resolved via
workspace.get_llm(profile_name=...)
. • #223 "capture active LLM profile from the correct users/me field" (merged). •
openhands/automation/utils/model_profiles.py
validates/resolves one profile name per automation. There is no
FallbackStrategy
/
fallback_llms
anywhere in the repo. Does extensions repo have a fallback PR? No LLM-fallback PR either. Related but not fallback: • #547 "show LLM provenance in automation output" (open) — resolves
AUTOMATION_MODEL
and prints
LLM profile · Model
. • #483 ships the reviewer script as a catalog bundle; #388 documents
AUTOMATION_MODEL
. • The pr-review plugin builds a single
LLM(**llm_config)
from
LLM_MODEL
with no
fallback_strategy
. Your pr-review automation In this environment's
automations.db
, the PR reviewer automations are single-profile with no fallback: •
GitHub PR Reviewer: OpenHands/software-agent-sdk for @all-hands-bot
→
gpt-5.6-sol
•
...OpenHands/OpenHands...
→
gpt-5.6-sol
•
...OpenHands/extensions...
→
None
(default) • multi-repo one →
deepseek-v4
So nothing is currently configured to fail over to another model. How to configure a fallback (e.g. to glm) At the SDK level it's a
FallbackStrategy
on the `LLM`:
Copy code
from openhands.sdk.llm import LLM, FallbackStrategy

llm = LLM(
    model="gpt-5.6-sol",
    api_key=...,
    fallback_strategy=FallbackStrategy(
        fallback_llms=["glm"],
        profile_store_dir="~/.openhands/profiles",  # defaults to this if omitted
    ),
)
• Profiles are JSON files in
~/.openhands/profiles
(or a custom dir). A checked-in glm-5.2 example already exists in the SDK at
examples/01_standalone_sdk/37_llm_profile_store/profiles/glm.json
—
"model": "openhands/glm-5.2"
with
usage_id: "glm"
. Drop that JSON (with a real key) into your profiles dir as
glm.json
. •
fallback_llms
takes an ordered list of profile names, so
["glm"]
will resolve that profile lazily through
LLMProfileStore
on the first failure. Two caveats for your specific case: 1.
fallback_strategy
is
exclude=True
on
LLM
, so it is not persisted/serialized — it must be re-attached after loading a profile (relevant for the automation path, which loads LLMs by profile name rather than constructing them inline). 2. Encrypted profiles aren't yet wired for fallback — bug #4558 (still open, PR #4568 "decrypt encrypted fallback profiles" is open/unmerged). If profiles are stored encrypted (server has
OH_SECRET_KEY
),
FallbackStrategy
currently resolves the fallback without the cipher, so the glm key arrives as Fernet ciphertext (
gAAAAA…
) and auth fails silently (the primary error gets re-raised). Until #4568 lands, plaintext glm profiles work, encrypted ones don't. Because the pr-review plugin hardcodes
LLM(...)
with no fallback and the automation service stores only a single
model
name, wiring "fallback to glm" end-to-end currently needs a small change on one side: either construct
FallbackStrategy
in
plugins/pr-review/scripts/agent_script.py
, or teach the automation service to accept/store a fallback list alongside
model
. There's no existing PR doing that yet.
c
@fresh-machine-37269 review 4568 and feel free to approve or request changes or comment. also link it for me here
👀 1
f
⏳ Still working — this is taking longer than this run allows. Follow along at the link above; mention me again for an update.