<@U0BME18CF7S> please review fixes for tests that...
# automations-debug
c
@fresh-machine-37269 please review fixes for tests that fail on every
main
run. the PRs for #17872 #17555, #17876 and #17889. For each, feel free to approve, request changes or comment, as you think appropriate. (openhands/openhands)
šŸ‘€ 1
f
šŸ¤– On it! View progress here: oss-agent-canvas.ngrok.dev/conversations/…
āœ… Done! Reviewed all three PRs and left a verdict on each: #17555 — āœ… Approved. The diagnosis is right: the old oracle polled the agent-server's global
/api/bash/bash_events
stream, which only holds commands the UI runs itself, so any exit-0
BashOutput
(even
stdout: null
) satisfied it. Scoping success to a
TerminalObservation
for the exact command, with the token required in that same observation, is the correct fix, and the 14 unit cases cover the tricky paths. Answering the SDK's title-generation request outside the scripted trajectory (mirroring the pre-flight ping) is a solid companion fix. Only ask: the positive Pull/Push assertion in
mock-llm-files-and-git.spec.ts
is a product-behaviour claim — make sure the maintainer decision behind it is visible on the PR. #17876 — šŸ” Changes requested. The fixture fixes are sound (recognizing the system + one-token user
ping
validation format, and writing npm-path user skills under
dirname(STATE_DIR)/skills
). But removing all six quarantines is pre-emptive: the PR's own evidence shows the full Linux workflow still exposes four tracked failures, so this just turns
skipped
into
failed
on
main
until the dependencies land. Also,
Validate PR description
is red here. #17889 — āœ… Approved. This is a real production fix, not a test-only restore: onboarding built the new profile from the dirty-field diff, dropping an endpoint/API key that already matched saved settings and silently falling back to the default provider (the
LLMAuthenticationError
in the recording). Reading the full saved settings and preserving unchanged values is right. Two notes:
Validate PR description
is red (needs the
bug
label on #17884), and the trajectory-activation overlap with #17876 should be spelled out. One clarification: #17872 is the tracking issue, not a PR, so there was nothing to approve there — it's best left open until all five failing specs actually pass on
main
. Also worth flagging: #17555 and #17876 both edit
tests/e2e/mock-llm/scripts/mock-llm-server.py
(title handling vs. validation handling), and both #17876/#17889 touch mock-LLM trajectory activation — so the merge order matters to avoid churn.
c
@fresh-machine-37269 please also review github.com/OpenHands/OpenHands/pull/18209 and similarly, approve it if you see no major issues
šŸ‘€ 1
f
āœ… Done! Reviewed PR #18209 and approved it. šŸ‘ It's a tidy, low-risk harness change confined to
.agents/skills/verify-openhands/
(plus
.pr/
evidence): the new pure helper
lib/agent-profile-repoint.mjs
decides when
llm preset
should repoint the seeded
default
agent profile, with unit tests covering the move, the already-there/missing-ref cases, and the ACP case. It's wired into
preset
only —
llm set
, named profiles, and fresh-run seeds are deliberately left alone, which matches the rationale in the description. All CI checks are green (the earlier "Validate PR description" failure was re-run and now passes), and I confirmed the helper's behavior against a few edge cases myself. No major issues found.
c
@fresh-machine-37269 thanks! same for github.com/OpenHands/OpenHands/pull/18211 please
šŸ‘€ 1
f
āœ… Done! Done — reviewed and approved PR #18211. šŸ‘ Same as before: it's a self-contained addition to the
verify-openhands
harness (no product code) that adds a
browser record start|pause|resume|stop|status
verb, backed by a new
lib/recording.mjs
with unit tests, plus help/SKILL/report updates. I re-ran the pure timeline and encoder-selection logic myself and confirmed
frameDurations
,
withoutPauses
,
frameRepeats
,
canvasSize
,
encoderSupport
, and the Playwright-ffmpeg lookup all behave as documented, and that the MP4/WebM/GIF paths and
pipe:0
stdin are correct. CI is green (the earlier PR-description check was re-run and now passes), and I saw no major issues.
c
@fresh-machine-37269 also github.com/OpenHands/OpenHands/pull/18207 please, the same
šŸ‘€ 1
f
āœ… Done! Done — reviewed and approved PR #18207. šŸ‘ It's a two-row docs update to the verify-openhands F07 feature map (plus
.pr/
evidence), no product code. I confirmed the pieces line up: • #18107 is merged, so the old
#18089
"Known failure" row correctly becomes an "Expected" row — prompt/agent hooks now show their text under Commands. • The new
#18206
row records a genuine, still-open gap:
hook-matcher-content.tsx
uses
hook.command || hook.prompt || hook.system_prompt
, so an agent hook carrying both fields shows the
prompt
instead of its
system_prompt
. • All the test IDs the updated recipe relies on (
hooks-modal
,
refresh-hooks
,
close-hooks-modal
,
show-hooks-button
,
conversation-name-context-menu
) exist in the source, and the "dialog closed first" precondition is a sound fix for the previous click-that-times-out. CI is green and the PR-description check passes. No major issues found.