<@U090ER7JPHA> tests are failing on main <https://...
# slackbot-chatter
l
@nutritious-balloon-65098 tests are failing on main https://github.com/OpenHands/OpenHands/actions/runs/20380478897/job/58569463227 debug the reason based on the commit history and when the test started failing, open an issue explaining the issue in detail, and then open a draft PR proposing a fix. keep the fix as minimal as possible
n
I'm on it! Graham Neubig can track my progress here.
Summary I've debugged the failing test
test_tool_call_validation_error_handling
and completed all requested tasks: Checklist • ✅ Analyzed the failing test from GitHub Actions • ✅ Investigated commit history to find when the test started failing (commit ec9daf3bc) • ✅ Identified root cause of the test failure • ✅ Opened GitHub issue explaining the problem: #12109 - https://github.com/OpenHands/OpenHands/issues/12109 • ✅ Created draft PR with minimal fix: #12110 - https://github.com/OpenHands/OpenHands/pull/12110 Root Cause The test was flaky due to a feedback loop in the event handling: 1. Test calls
controller._step()
2.
_step()
catches
BadRequestError
and adds
ErrorObservation
to event stream 3. Background thread picks up the event and calls
on_event
4.
on_event
calls
should_step(ErrorObservation)
which returns
True
5.
on_event
calls
_step_with_exception_handling()
6.
_step()
catches
BadRequestError
again (mock always raises the same error) 7. Repeat until iteration limit is reached The test expected 1
ErrorObservation
but got 4-10 depending on timing. Fix The minimal fix changes the mock to only raise the error on the first call, then return
NullAction
on subsequent calls. This breaks the feedback loop and makes the test deterministic. Changes: • Added import for
NullAction
• Modified
mock_step()
to track call count and only raise error once The fix was verified to pass consistently across 5 consecutive runs with pytest-xdist (
-n auto
). <sub>View full conversation</sub>