Fixing a flaky test <https github com OpenHands agent canvas OpenHands #proj-agent-canvas

Fixing a flaky test: <https://github.com/OpenHands...

modern-fireman-99755

07/02/2026, 8:07 AM

Fixing a flaky test: https://github.com/OpenHands/agent-canvas/pull/1580 The problem was that

activateProfileViaUI

(the setup step every mock-LLM e2e test runs) flaked because its 15s poll did a full page.goto("/settings/llm") reload on every tick, unnecessary, since the "Active" badge updates reactively, and it could silently skip the activation click when the just-created profile row hadn't rendered yet, then wait for a badge that was never set. The fix is to wait for the row to appear, retry the open-menu → "Set active" gesture with expect().toPass(), and assert the badge shows reactively — no per-tick reload. This has the same coverage as a test. cc @quick-window-91705

modern-fireman-99755

07/02/2026, 11:47 AM

Actually, there it was another flaky test which pointed to a real problem https://github.com/OpenHands/agent-canvas/pull/1582 Brief description of what happened: The onboarding "backend connected" check was flaking in CI. The health probe only re-checks every 10s and doesn't retry a failed attempt, so if the very first probe misses (agent-server still warming up right after page load), the "connected" badge sits there for a full 10s and the test times out. This adds a quick retry (2×300ms) so a transient first miss recovers in under a second instead of waiting for the next poll. I ran the onboarding e2e locally against the mock stack and it connects and completes fine; also a nice side effect — real users on a slow/cold backend see "connected" sooner. If you could review also this @quick-window-91705 could be fantastic excited

✅ 1

modern-fireman-99755

07/02/2026, 1:04 PM

Copy code

Workflow                         Period            Runs (pass+fail)  Failures  Rate
Mock-LLM E2E Tests (Playwright)  May 27 – Jul 2    1,187             305       26%
Mock-LLM Docker E2E Tests        Jun 1 – Jul 2     1,661             290       17%
Agent E2E Selector               Jun 8 (10 runs)   9                 6         —

modern-fireman-99755

07/02/2026, 1:04 PM

A small statistic of rate of failures we have (it is just en estimation). We don't know how much is because of flaky tests or because the test failed correctly.

Open in Slack

Previous Next