I am trying to run the SWE benchmark on the openha...
# questions
t
I am trying to run the SWE benchmark on the openhands-benchmark repo. When i build the images and then run the tests the coding agent is unable to get scikit working. I suspect that the problem is because the agent_server image is being built with python 3.13. I changed the docker to build with 3.11 but this seems like a sloppy fix. Any thoughts ?
m
Hey @thankful-dusk-15577 what do you mean by "_the coding agent is unable to get scikit working" ?_
t
it cant get the library to work / load
m
Could you share logs? To understand what is going on...
t
Yes, sorry i deleted the logs let me rebuild and rerun
ok now its working, its explicitly using the conda environment now which it wasnt before. Maybe just an agent mistake before...
👍 2
m
Cool :) Just out of curiosity: What was the model you were using?
t
qwen 3.6 27b.
i have some rate limiting on top of it so the connection / tests are a bit unstable
m
thanks for sharing.
t
ah it did it again. Here it seems to be falling apart
Reasoning: The build is failing due to compatibility issues between the Python version and the old scikit-learn version. However, since Pipeline is pure Python, I can bypass the __check_build by directly importing from the module file. Let me try a different approach - I can bypass the check_build import or directly test the pipeline module. Actually, let me just look at the pipeline.py file directly. Since this is a pure Python file, I don't need to build the entire project to make my changes. I can: 1. Read and understand the pipeline.py file 2. Make the changes 3. Test by directly importing just the pipeline module Let me also check if there's a way to bypass the check_build.
👀 1
m
Ok I checked and the problem is in the prompt we are giving to the model in swebench. For more info see: https://github.com/OpenHands/benchmarks/issues/753 We never saw it before (or just partially) because the model could resolve the problem alone. For example, gemini-flash-3.5 did
Copy code
"…the path to the conda.sh script, ./opt/miniconda3/..., looks a bit ambiguous regarding its absolute or relative nature… my instinct is to verify… whether the absolute path /opt/miniconda3/etc/profile.d/conda.sh is the correct one."
It then ran source /opt/miniconda3/etc/profile.d/conda.sh && conda activate testbed, got testbed * active, and proceeded normally.
🙌 1