https://docs.all-hands.dev/ logo
Join Slack
Powered by
Hello, I am wondering for benchmarking, is there a...
# questions
a
Hello, I am wondering for benchmarking, is there a way to use Podman/Apptainer instead of Docker? I am using my school's server and they do not allow docker. Thanks!
l
a
Oh, great. May I ask how does the software-agent-sdk interact with the benchmark repo? I am new to OpenHands and am not too familiar with the code base. Thanks!
l
@nutritious-balloon-65098 explain how apptainer can be used with the OpenHands/benchmarks repo, and if it is not properly documented in the repo send a PR to update
n
I'm on it! Graham Neubig can track my progress here.
Done. ### Final summary Answer: Apptainer is supported by the vendored OpenHands SDK, but not yet as a built-in 
--workspace apptainer
option in the 
OpenHands/benchmarks
repo. In this repo today, benchmark runners only support 
docker
and 
remote
. ### Checklist - [x] Verified how Apptainer fits into the stack - [x] Confirmed the benchmarks repo did not document this clearly - [x] Updated the docs to explain the current state - [x] Kept the changes concise - [x] Validated the edited files with pre-commit - [x] Pushed the branch and opened a PR ### What changed Only two documentation files were updated: - 
README.md
- 
AGENTS.md
The README now explains: - the SDK has 
ApptainerWorkspace
- Apptainer is useful on Docker-restricted systems - the benchmarks CLI currently only exposes 
docker
and 
remote
- current options are either: 1. use 
--workspace remote
, or 2. add local benchmark-side integration for 
ApptainerWorkspace
### Conciseness check The changes were documentation-only and targeted to your request. No benchmark logic or runtime behavior was modified. ### PR - PR #509: https://github.com/OpenHands/benchmarks/pull/509 If you want, I can next draft the actual code changes to add 
--workspace apptainer
support to one of the benchmark runners.
l
@nutritious-balloon-65098 yes do that.
@quick-window-91705 do follow-up messages no longer work?
q
uh oh, ill patch this
a
Thanks for the help! Let me know when the PR is ready and I can help test it out
n
Done — the request has been completed and the changes were pushed. Checklist: - [x] Implemented real benchmark-side 
--workspace apptainer
support - [x] Updated supported runners to use 
ApptainerWorkspace
- [x] Added focused tests for the new workspace path - [x] Updated docs to explain Apptainer usage and its limitations - [x] Validated with 
pre-commit
and targeted 
pytest
- [x] Pushed the changes to the remote branch - [x] Updated the existing PR instead of opening a new one Conciseness check: - The committed changes were limited to the Apptainer feature, related docs, and targeted tests. - A local 
AGENTS.md
memory note remains unstaged and was intentionally not included in the commit/PR, so no extraneous repo changes were pushed. Final summary: - Added benchmark-side support for 
--workspace apptainer
in the shared CLI/model layer. - Added a reusable 
create_apptainer_workspace()
helper for pulling/running pre-built agent-server images via Apptainer. - Wired Apptainer support into GAIA, Commit0, MultiSWEBench, SWE-Bench, SWTBench, SWE-Bench Multimodal, SWE-fficiency, and OpenAgentSafety. - Documented that Apptainer is intended for Docker-restricted environments and requires pre-built registry images rather than local on-the-fly Docker builds. - Added tests covering the new helper and parser support. - Pushed commit 
8d6113e
to branch 
docs/apptainer-benchmark-clarification
. PR: - https://github.com/OpenHands/benchmarks/pull/509 No further changes were made after that push.
a
Apology for the delayed response, I got caught up in school work. I have tested out the PR and did not go as expected, here are the commands that I run:
Copy code
uv run python -m benchmarks.swebench.build_images \
  --dataset princeton-nlp/SWE-bench_Lite \
  --split test \
  --image <http://ghcr.io/openhands/eval-agent-server|ghcr.io/openhands/eval-agent-server> \
  --target source-minimal

uv run swebench-infer .llm_config/vllm.json \
    --dataset princeton-nlp/SWE-bench_Lite \
    --split test \
    --max-iterations 200 \
    --num-workers 12 \
    --workspace apptainer
and I got the following error:
Copy code
Instance sympy__sympy-24909 - 2026-03-15 16:03:42,777 - ERROR - [child] Instance sympy__sympy-24909 failed after 3 retries. Last error: Agent server image <http://ghcr.io/openhands/eval-agent-server:bde715c-sweb.eval.x86_64.sympy_1776_sympy-24909-source-minimal|ghcr.io/openhands/eval-agent-server:bde715c-sweb.eval.x86_64.sympy_1776_sympy-24909-source-minimal> does not exist in container registry. Apptainer workspace requires a pre-built image that can be pulled with Apptainer.
Traceback (most recent call last):
  File "/n/home08/hzeng/benchmarks/benchmarks/utils/evaluation.py", line 751, in _execute_single_attempt
    workspace = self.prepare_workspace(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home08/hzeng/benchmarks/benchmarks/swebench/run_infer.py", line 190, in prepare_workspace
    workspace = create_apptainer_workspace(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home08/hzeng/benchmarks/benchmarks/utils/image_utils.py", line 113, in create_apptainer_workspace
    raise RuntimeError(
RuntimeError: Agent server image <http://ghcr.io/openhands/eval-agent-server:bde715c-sweb.eval.x86_64.sympy_1776_sympy-24909-source-minimal|ghcr.io/openhands/eval-agent-server:bde715c-sweb.eval.x86_64.sympy_1776_sympy-24909-source-minimal> does not exist in container registry. Apptainer workspace requires a pre-built image that can be pulled with Apptainer.
Would I need to build the images in a different way? Thanks!
l
@nutritious-balloon-65098 answer this question and make any fixes to the PR if any fixes are necessary
n
I'm on it! Graham Neubig can track my progress here.
Final summary: Answer: for 
--workspace apptainer
, the SWE-Bench images must be registry-pullable or already present as a cached SIF. A local 
build_images.py
run without 
--push
 is not sufficient, which is why your run failed. ### Checklist - [x] Confirmed the reported behavior and identified the root cause - [x] Verified the request was fully addressed - [x] Kept the follow-up changes focused on the Apptainer issue only - [x] Avoided extraneous repo changes in the commit ### New changes since the last summary - Added a small runtime improvement so Apptainer can reuse an existing cached SIF from 
APPTAINER_CACHE_DIR
instead of always requiring a registry lookup. - Improved the Apptainer error message to explicitly say that local-only builds are not enough and that 
build_images.py
should be re-run with 
--push
. - Updated the top-level README and SWE-Bench README to clearly document the required Apptainer workflow: 1. build on a Docker-capable machine or CI, 2. use 
--push
, 3. run inference with 
--workspace apptainer
, 4. optionally reuse cached SIFs afterward. - Added targeted test coverage for the cached-SIF path and updated the helper tests accordingly. ### Conciseness check - The committed changes are limited to: - 
benchmarks/utils/image_utils.py
- 
tests/test_image_utils.py
- 
README.md
- 
benchmarks/swebench/README.md
- A local 
AGENTS.md
memory note remains unstaged and was intentionally not included in the commit. ### Validation - 
uv run pre-commit run --files benchmarks/utils/image_utils.py benchmarks/swebench/README.md README.md tests/test_image_utils.py
- 
uv run pytest tests/test_image_utils.py -q
Both passed. ### Commit - 
2f23853
— 
Improve Apptainer image guidance and cache reuse
No further changes were made after that commit.
l
@busy-branch-28834 I know you’re running things in apptainer, do you have any comments here?
n
Final summary of work: ### Checklist - [x] Pushed the follow-up code/doc fix to the existing PR branch: 
2f23853
- [x] Added a PR comment explaining that Apptainer requires registry-pullable images built with 
--push
- [x] Updated the main PR description with an Evidence section - [x] Included the required statement: “There is still insufficient evidence to merge this PR, since it has not been run end-to-end.” - [x] Set PR #509 back to draft - [x] Attempted an end-to-end validation run in this environment - [x] Documented that end-to-end Apptainer validation is blocked by human QA ### End-to-end validation attempt I tried this minimal run:
Copy code
bash
uv run gaia-infer .llm_config/example.json --workspace apptainer --n-limit 1 --num-workers 1 --max-iterations 1 --max-attempts 1
Result: - it did not reach the Apptainer workspace - it failed earlier because the GAIA dataset is gated, and the available Hugging Face token in this sandbox returned 403 Forbidden Additional environment blockers here: - 
apptainer
is not installed - 
/dev/fuse
is unavailable - 
/var/run/docker.sock
is unavailable ### Conciseness check - The only new repo change pushed was the focused follow-up commit already prepared for this PR branch. - The rest of the work was limited to PR metadata updates: comment, description, and draft status. - No extra unrelated repository changes were pushed. ### PR - PR #509: https://github.com/OpenHands/benchmarks/pull/509
Final summary of the follow-up work: ### Checklist - [x] Re-tried validation using a public dataset (
princeton-nlp/SWE-bench_Lite
) instead of gated GAIA - [x] Found a real published agent image for 
astropy__astropy-12907
- [x] Ran a minimal one-instance Apptainer eval - [x] Confirmed the PR’s logic gets through dataset access + image resolution and reaches 
ApptainerWorkspace
initialization - [x] Confirmed the remaining blocker in this sandbox is the runtime environment, not the benchmark codepath - [x] Updated PR #509: - PR comment - PR description Evidence section - kept the PR in draft - [x] Made no additional code changes in this follow-up ### Result The public SWE-Bench attempt was stronger evidence than the earlier GAIA run: - It successfully used a public dataset - It resolved a published image: - 
<http://ghcr.io/openhands/eval-agent-server:bde715c-sweb.eval.x86_64.astropy_1776_astropy-12907-source-minimal|ghcr.io/openhands/eval-agent-server:bde715c-sweb.eval.x86_64.astropy_1776_astropy-12907-source-minimal>
- It reached 
ApptainerWorkspace
- It then failed immediately with: - 
[Errno 2] No such file or directory: 'apptainer'
### Conclusion The follow-up did not reveal a new PR code bug. It validated that the PR’s Apptainer image-selection path works for a public SWE-Bench instance, and that the remaining blocker here is the sandbox environment: - 
apptainer
not installed - 
/dev/fuse
unavailable - no Docker socket available for fallback ### Conciseness check - No new source changes were made - No new commit was needed for this follow-up - Only PR metadata/comment updates were made, plus local evaluation artifacts for evidence ### Remote branch status The previously prepared code changes were already pushed earlier on branch 
docs/apptainer-benchmark-clarification
in commit: - 
2f23853 Improve Apptainer image guidance and cache reuse
PR: - https://github.com/OpenHands/benchmarks/pull/509 So the final answer is: the PR has been updated with stronger evidence, but there is still insufficient end-to-end evidence to merge because Apptainer cannot actually run in this sandbox.
b
Hi Graham sir and @abundant-librarian-24982! I can think of 3 possible reasons why we are getting this error: 1. [Most likely] The build command 
uv run python -m benchmarks.swebench.build_images
will only build the docker images locally and not push them to the registry. When you run evaluation with 
--workspace apptainer
flag, it internally runs the command 
apptainer pull <path to .sif file> <http://ghcr.io/|ghcr.io/><image_name>
and it may be the case that this command doesn't check for local docker cache. Or even worse, if you build the image in a different machine and ran eval in other machine, this will obviously fail. The fix will be to push these images to your personal DockerHub account or in some other public container registry. 2. [Slightly unlikely but maybe useful] Make sure to authenticate yourself by running 
apptainer registry login
(see here) to avoid hitting rate limit errors to DockerHub. 3. [unlikely] Network issues: If the cluster is having network issues then this may happen as well
a
ok, got it. Let me retry. I am also wondering about the setup of the evaluation. Why do we need to build the images? From the source code I see that we are just checking if online has that version and skip if it does. Wouldn't benchmarks' images always be there?
b
I don't know if benchmarks' developers have pushed these images to a registry and whether they are publicly accessible or not. @millions-crayon-32880 can answer this question better I guess
a
Ah, ok. Another question, I see the following error after adding --push tag: [stderr] Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg. [stderr] Error: unknown flag: --push [stderr] See 'podman buildx build --help' my school's HPC does not allow docker, so they have podman instead. I assume this would mean that my push command did not get carried out successfully
b
The build images parser for swe-bench has a push flag for sure, see here: https://github.com/OpenHands/benchmarks/blob/e326a67d763f5f2e0a46d4df2a25826bb56771ca/benchmarks/utils/build_utils.py#L340. I don't have much idea about the specifics of podman but you will also need to rename the images to point to your own registry (you can't just try and push to ghcr.io/openhands... since you don't have access). This means editing the value of the 
EVAL_AGENT_SERVER_IMAGE
here: https://github.com/OpenHands/benchmarks/blob/e326a67d763f5f2e0a46d4df2a25826bb56771ca/benchmarks/utils/constants.py#L7 and point it to your registry.
2 Views
Open in Slack
PreviousNext
Powered by