https://docs.all-hands.dev/ logo
Join Slack
Powered by
Got the new Gemma 4 model working on my macbook us...
# random
a
Got the new Gemma 4 model working on my macbook using the CLI - details in the thread.
🙌 3
This is the first local model i have used with OpenHands, this is what I did: 1. Install llama.cpp (needs HEAD build for Gemma 4 support) 
brew install llama.cpp --HEAD
2. Start the server — it auto-downloads the model from HuggingFace 
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF:Q8_0
3. Point OpenHands CLI at it
Copy code
LLM_MODEL="openai/gemma-4-it" \
LLM_BASE_URL="<http://127.0.0.1:8080/v1>" \
LLM_API_KEY="not-needed" \
openhands --override-with-envs
That was it and its working all on my laptop
❤️ 1
The generation speed is rock-solid — every recent request is within a hair of 20 tokens/sec. Prompt eval varies a bit more (348–407 tok/s) depending on how much of the KV cache can be reused.
h
Nice! I'm going to give it a try tonight
a
I also found this when I was looking around, we should just make it easy to use these models: https://github.com/OpenHands/OpenHands/pull/13532
h
I don't follow how that works. But I do agree on making it easier for local support!
nice clear step by step
a
The llama-server approach above worked really easy on my laptop it was literally those three steps
I also had openhands CLI running and it would check the download status, logs, and let me know how it was all running
h
Alternatively with the ollama app its just download with the app and put the values in the CLI and reload it
a
nice, which gemma4 did it do, since there are several
h
So e4b
👍 1
2 Views
Open in Slack
PreviousNext
Powered by