Using (small) local models with OpenHands We have servers with 6GB GPUs that I would like to use local models to generate boilerplate code. The servers are powerful enough to run Aider with small local models, but OpenHands malfunctions. We checked https://github.com/OpenHands/OpenHands/issues for similar issues but did not find one. We also read the OpenHands documentation related to using local models. I suspect others have had issues with running (small) models locally. Models we've tried: • devstral:24b • deepseek-r1:14b • deepcoder:14b • mistral:7b • deepseek-coder:6.7b • wizardcoder:7b-python • qwen3:8b • qwen3-coder:30b ModelSizeStatusdevstral24b ✅ Works deepseek-r114b ❌ Gets stuck in loop deepcoder14b ⚠️ Creates file but gets stuck mistral7b ❌ Doesn't work deepseek-coder6.7b ❌ Doesn't work wizardcoder7b-python ❌ Doesn't work qwen38b ⚠️ Somewhat worked qwen3-coder30b ❌ Doesn't work The smaller models either didn't work or only somewhat worked. If I recall correctly it was qwen3:8b that somewhat worked. Being larger in size didn't necessarily mean the model worked, but of the models that did the job and didn't encounter issues, they were 14b+ ones. The primary issue I recall encountering was the models not understanding that a back and forth conversation was taking place. They didn't understand:

> Please continue on whatever approach you think is suitable. 
  If you think you have solved the task, please finish the interaction.

There is also the models not knowing about or having the ability to interact with local files through the agent, such as:

> I'm sorry for the confusion, but as an AI model developed by OpenAI, 
  I don't have access to a runtime environment or any specific hosts. 
  My main function is to provide information and answer questions related 
  to programming and computer science based on my training data.

Ollama

$ podman run -d \
    --device nvidia.com/gpu=all \
    -e OLLAMA_MODELS="/data/ollama/models" \
    -v /opt/mm/data:/data \
    -v /opt/ollama:/root/.ollama \
    -p 11434:11434 \
    --name ollama \
    --security-opt=label=disable \
    ollama/ollama
$ podman exec -ti ollama ollama pull deepseek-r1:14b
$ podman exec -ti ollama ollama pull deepseek-coder:6.7b
$ podman exec -ti ollama ollama pull devstral:24b
$ podman exec -ti ollama ollama pull deepcoder:14b
$ podman exec -ti ollama ollama list
NAME                   ID              SIZE      MODIFIED    
deepcoder:14b          12bdda054d23    9.0 GB    19 seconds ago
deepseek-coder:6.7b    ce298d984115    3.8 GB    4 days ago    
devstral:24b           9bd74193e939    14 GB     4 days ago    
deepseek-r1:14b        c333b7232bdb    9.0 GB    4 days ago

Headless OpenHands runs deepseek-r1:14b Gets stuck in loop

$ podman run -it \
     --pull=always \
     -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.openhands.dev/openhands/runtime:1.3-nikolaik \
     -e SANDBOX_USER_ID=$(id -u) \
     -e SANDBOX_VOLUMES="/opt/openhands/volume:/workspace:z" \
     -e LLM_BASE_URL="<http://localhost:11434>" \
     -e LLM_API_KEY=none \
     -e LLM_MODEL="ollama/deepseek-r1:14b" \
     -e LLM_TIMEOUT=2400 \
     -e LOG_ALL_EVENTS=true \
     -e RUN_AS_OPENHANDS=false \
     -e SANDBOX_USE_HOST_NETWORK=true \
     -e DEBUG=true \
     -v /var/run/user/1000/podman/podman.sock:/var/run/docker.sock:z \
     -v /opt/openhands/openhands:/.openhands:z \
     -v /opt/openhands/volume:/workspace:z \
     --network host \
     --add-host host.docker.internal:host-gateway \
     --name openhands-app-$(date +%Y%m%d%H%M%S) \
     --security-opt label=disable \
     docker.openhands.dev/openhands/openhands:1.3 \
     python -m openhands.core.main -t "Write fizz buzz in bash"

deepcoder:14b Gets stuck in loop but does create fizzbuzz.sh

$ podman run -it \
     --pull=always \
     -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.openhands.dev/openhands/runtime:1.3-nikolaik \
     -e SANDBOX_USER_ID=$(id -u) \
     -e SANDBOX_VOLUMES="/opt/openhands/volume:/workspace:z" \
     -e LLM_BASE_URL="<http://localhost:11434>" \
     -e LLM_API_KEY=none \
     -e LLM_MODEL="ollama/deepcoder:14b" \
     -e LLM_TIMEOUT=2400 \
     -e LOG_ALL_EVENTS=true \
     -e RUN_AS_OPENHANDS=false \
     -e SANDBOX_USE_HOST_NETWORK=true \
     -e DEBUG=true \
     -v /var/run/user/1000/podman/podman.sock:/var/run/docker.sock:z \
     -v /opt/openhands/openhands:/.openhands:z \
     -v /opt/openhands/volume:/workspace:z \
     --network host \
     --add-host host.docker.internal:host-gateway \
     --name openhands-app-$(date +%Y%m%d%H%M%S) \
     --security-opt label=disable \
     docker.openhands.dev/openhands/openhands:1.3 \
     python -m openhands.core.main -t "Write fizz buzz in bash"

devstral:24b This works

$ podman run -it \
     --pull=always \
     -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.openhands.dev/openhands/runtime:1.3-nikolaik \
     -e SANDBOX_USER_ID=$(id -u) \
     -e SANDBOX_VOLUMES="/opt/openhands/volume:/workspace:z" \
     -e LLM_BASE_URL="<http://localhost:11434>" \
     -e LLM_API_KEY=none \
     -e LLM_MODEL="ollama/devstral:24b" \
     -e LLM_TIMEOUT=2400 \
     -e LOG_ALL_EVENTS=true \
     -e RUN_AS_OPENHANDS=false \
     -e SANDBOX_USE_HOST_NETWORK=true \
     -e DEBUG=true \
     -v /var/run/user/1000/podman/podman.sock:/var/run/docker.sock:z \
     -v /opt/openhands/openhands:/.openhands:z \
     -v /opt/openhands/volume:/workspace:z \
     --network host \
     --add-host host.docker.internal:host-gateway \
     --name openhands-app-$(date +%Y%m%d%H%M%S) \
     --security-opt label=disable \
     docker.openhands.dev/openhands/openhands:1.3 \
     python -m openhands.core.main -t "Write fizz buzz in bash"

$ ls -Al /opt/openhands/volume/
total 4
drwxr-xr-x. 2 elrod elrod   6 Jan 14 19:53 .downloads
-rwxr-xr-x. 1 elrod elrod 284 Jan 14 19:54 fizzbuzz.sh

Thank you for the feedback! I'd make a few quick notes • it seems you're running OpenHands V0, the legacy version. OpenHands V1 has been rebuilt a lot. This might not make a big difference on this particular issue, though, sadly. Just for clarity. • This "The primary issue I recall encountering was the models not understanding that a back and forth conversation was taking place." sounds like an LLM issue to me. IIRC Aider is single step or close to? • This "I don't have access to a runtime environment" is probably the same (and some LLMs can be convinced) • There are small models that work with OH, as you noted These are just super-quick notes, sorry for not addressing a lot

Another common issue is that default context size is usually to small to load system prompt + skills/tools. If you increase to fit then inference performance can be bad, especially for those models bigger than your GPU. Aider has a small system prompt and no skills/tools which helps in these cases.

The issue regarding the back and forth conversation breaking things can be fixed by removing a line in the jinja chat template. I had to do this for my local Devstral to work with OpenHands.