<@U0BME18CF7S> /codereview this PR <https://github...
# automations-debug
c
@fresh-machine-37269 /codereview this PR https://github.com/OpenHands/software-agent-sdk/pull/4921#issuecomment-5613156811 ; find the reason why the bot failed, it was the pr review automation in your environment. then look at automation repo and openhands repo for the similar PRs; review those too.
πŸ‘€ 1
f
πŸ€– On it! View progress here: oss-agent-canvas.ngrok.dev/conversations/…
βœ… Done! I have everything I need. Here's the full picture. Why the bot failed The reviewer automation didn't fail because of the PR β€” it failed on LLM quota exhaustion in your environment. The run logs (
agent-canvas.2026-09-10.log
, run
0ba1fb10…
) show the review conversation erroring out with:
Copy code
litellm.RateLimitError: {"error":{"type":"usage_limit_reached",
  "message":"The usage limit has been reached","plan_type":"team",
  "resets_in_seconds":421890}}
The agent retried 4Γ— (backing off each time), the conversation went to
error
, and the automation posted the generic
⚠️ …encountered a problem (status: error)
comment.
resets_in_seconds β‰ˆ 421890
β‰ˆ ~4.9 days, so it will keep failing until the team-plan limit resets or is raised β€” unless a fallback model kicks in. Notably, the fix for exactly this is sitting unmerged in this same repo:
software-agent-sdk#4917
"Fail over to fallback LLM immediately on hard quota exhaustion"
(fork branch
quota-fallback-immediate
). It adds
is_quota_exhaustion_error()
so
usage_limit_reached
/
insufficient_quota
skip the retry loop and fail over to a fallback LLM on the first attempt instead of after the full backoff. I reviewed that diff too β€” it's clean and correct (a
retry_if_exception_type(...) & retry_if_exception(lambda e: not is_quota_exhaustion_error(e))
tenacity predicate). Merging it only helps if a fallback model is actually configured; the automation's
model:
is currently
null
. The same failure hit
OpenHands#17221
(the identical PR in the OpenHands repo) at 04:33 β€” same quota error.
automation#431
and
extensions#451
are still pre-review (HUMAN-section/CI gates), so they haven't spent an LLM call yet. The PR itself (software-agent-sdk#4921) Docs-only: adds
.agents/skills/pr-design-doc/SKILL.md
(168 lines) +
references/html-craft.md
(322 lines). This is a skill that teaches the agent to author a self-contained HTML design doc for non-trivial PRs. Low risk, well written. The workflow (verify PR head β†’ read both sides β†’ before/after SVG β†’ commit under
.pr/
β†’ link via htmlpreview) and the safety stance (reject symlinks, never publish private-repo docs, escape all source-derived text) are genuinely good. Issues worth raising (non-blocking): 1. Duplicated across four repos β€” this is the real smell. The exact same skill is landing in
software-agent-sdk
,
OpenHands
,
automation
, and already exists in
extensions#451
. All three new ones have a byte-identical
SKILL.md
(blob sha
62f02c6a…
) and near-identical
html-craft.md
.
extensions
is the canonical skills home β€” this copy-paste will drift (it's already drifted: 168 vs 137 lines SKILL, 322 vs 295 lines html-craft). Recommend one source of truth, with the others referencing it. 2.
test ! -L .pr && test ! -L .pr/design.html
is a reasonable guard but is advisory, not a security boundary β€” it's a TOCTOU check on a worktree the agent already owns. Fine, just don't over-claim. 3. Cleanup assumption is repo-specific. The skill correctly says to verify
.github/workflows/pr-artifacts.yml
exists before relying on cleanup. I confirmed it exists in
software-agent-sdk
(and in `OpenHands`/`automation` checkouts), and its
cleanup-on-approval
job only runs for same-repo PRs β€” for forks it deletes
.pr/
from base after merge. The skill's "fork PR β†’ remove manually before merge" instruction matches, so this is consistent, but it means the doc can linger on fork PRs. 4. Trigger collision risk:
/design-doc
is a very generic trigger; combined with the long
description
keywords it may fire more broadly than intended (e.g. "add a design doc"). Worth tightening. Verdict: βœ… approve (as docs) once the maintainer decides on the dedup strategy; the bot failure itself is an infra/quota issue, not a code issue. The sibling PRs β€’
OpenHands/OpenHands#17221
β€” same content, same author (enyst),
Fixes #16304
. Same review applies verbatim. Currently stuck on the same quota error. β€’
OpenHands/automation#431
β€” same content (
Fixes
an automation issue). Bot has only posted the HUMAN-section nudge so far, not reviewed. β€’
OpenHands/extensions#451
β€” the original (smolpaws), in the older
skills/<name>/
plugin/marketplace layout (with `.claude-plugin`/`.codex-plugin`/`commands/`) rather than the newer
.agents/skills/
convention. This is the one the others should arguably consolidate into. Currently pre-review (HUMAN + CI gates). Recommended next step: merge
#4917
(the quota-failover fix) and/or raise/reset the OpenAI team quota, then re-trigger the reviews on
#4921
,
#17221
, and
#431
.
c
@fresh-machine-37269 It's OK, we want to make them repo skills, not generic skills. You can find the reasoning in the comments on the original smolpaws PR. They are not generic, they wouldn’t apply to private repos and other conditions. Feel free to approve them properly. Link the failover fix.
πŸ‘€ 1
f
βœ… Done! All three approved. βœ… Approvals submitted (as
all-hands-bot
, event
APPROVE
): β€’
OpenHands/software-agent-sdk#4921
β€” review id 5163028812 β€’
OpenHands/OpenHands#17221
β€” review id 5163029280 β€’
OpenHands/automation#431
β€” review id 5163029690 I confirmed the reasoning you referenced: Engel's
CHANGES_REQUESTED
review on
extensions#451
(the original smolpaws PR) says to keep the skill repo-local rather than in the global extensions registry, because the
.pr/
cleanup/hosting contract is repo-specific β€” which is why the corrected copies landed in
OpenHands
,
software-agent-sdk
, and
automation
as
.agents/skills/
rather than the generic catalog. That addresses my earlier "duplication" concern. Failover fix link: github.com/OpenHands/software-agent-sdk/pull/4917 β€” "Fail over to fallback LLM immediately on hard quota exhaustion" (still open, authored by
all-hands-bot
). This is the fix for the quota-exhaustion failure that caused the reviewer to error out on `#4921`/`#17221`. Note it only helps if a fallback LLM is actually configured β€” the reviewer automation's
model:
is currently
null
, so the quota limit still needs to be reset/raised for the bot to recover now.