Hi everyone! I have a question regarding running the full SWE-Bench Verified evaluation (500 instances) using OpenHands. So far I've noticed that each instance Docker image for the sandbox environment is quite large (8-12GB each). After running only 34 instances (see attached image), image storage is already almost at 400GB. 1. Is this to be expected? 2. If so, are there recommended workflows for managing this? 3. If this isn't the expected behaviour, I'd also appreciate any guidance on how to best set it up. I'm running this on an Ubuntu 24.04 VM with OpenHands 0.58.0 (an older version and not using the SDK because I'd like to run experiments with Ambiguous/Interactive SWE Bench and some custom agents). Thanks for your help!

Hey @early-fireman-79321, thanks a bunch! Actually I was just speaking with @plain-jordan-64103 and @dry-spoon-69117 and we’d like to get ambiguous/interactive swe bench in the new eval repo as well: https://github.com/OpenHands/benchmarks Once we do that we’ll be able to run with the newer version of openhands, which (1) should make it much easier to experiment with different types of agents too, (2) should make docker container building much easier too. If you’re interested in this, maybe we can coordinate and try to get it in the new benchmarks ASAP.

ah i think i've run into this as well for swe-fficiency implementation on v1 benchmarks (both v0 and v1 had this limitation iirc): i think the runtime/workspace cleanup cleans up the container (but not the image necessarily) which is whats adding up as evaluation occurs filed a issue https://github.com/OpenHands/software-agent-sdk/issues/1297 https://github.com/OpenHands/software-agent-sdk/blob/main/openhands-workspace/openhands/workspace/docker/workspace.py#L336

Thanks @limited-student-10747! Would be very happy to collaborate on this with you guys, maybe we can have a call to discuss this further?

Hey @early-fireman-79321, cool! BTW does Jeff’s suggestion fix your issue though?

something like (the inverse of) https://github.com/OpenHands/OpenHands/pull/11716/commits/20ca65d166380c6f597f8ee11867b9106c0bb717#diff-e7bc132f7cd35153d2[…]6316869d3b6bf16b1defb16 was my quick bandaid fix to unblock you

Sorry for the delayed response - yes the suggestion from @jolly-orange-24543 did work, thank you!!

BTW @proud-twilight-98040 is this similar to what you were experiencing? If so, since it seems that multiple people are encountering this we should prioritize a proper fix.

Yeah, sounds like the same issue. Does @jolly-orange-24543's solution delete the images? I guess that means that subsequent evaluation runs have to re-build all the images.

yea, my solution deletes both the openhands image + base image after each runtime closes: realistically, this rebuilding needs to happen regardless for swebench evaluation (since each openhands runtime image is based on each instances swe-bench base image, so any new instance to eval will be a cache miss in terms of image) my solution doesn't clear the build cache though, so you should inherit reasonable build speedup still for the components that are common across swe-bench images (though the build cache will also add up as evaluation goes on since its not cleared) would be cool to see if theres an easy way to invert the building (i.e. have openhands in the base image and build SWE-bench on top for faster build times)