Hi everyone, I’m running into an issue with the latest OpenHands codebase while working on SWE-bench evaluation, and I’d appreciate your help. I have a local fork of OpenHands from May 2025 where SWE-bench evaluation works correctly. This month I cloned the latest version of the repository to migrate my changes, but I encountered a problem before applying any of my modifications. Specifically, when I run the SWE-bench inference command and provide max_iterations (e.g., 100 or even 1000), the system immediately runs through all iterations without performing any LLM calls or meaningful agent steps. Logging shows that the iteration loop executes exactly as many times as specified (e.g., 100 logs for max_iter=100), but no agent actions or LLM interactions occur. It seems like something is triggering the iteration loop to advance instantly, possibly related to recent changes in the event-driven execution model or the control-flag / iteration-limit logic. My May 2025 version does not exhibit this behavior on the same machine and configuration. If you could provide guidance on what might have changed in the iteration or event-triggering logic—or point me to the correct place to start troubleshooting—I’d greatly appreciate it. Thank you!

You should probably check out https://github.com/OpenHands/benchmarks @chilly-france-44439 should we delete the evaluation/ directory in OpenHands?