Pardon my spam, but I wanted to ensure that I get ...
# questions
t
Pardon my spam, but I wanted to ensure that I get a robust answer because I have run into this problem and I think it may be something I am doing wrong. Setup - Backend: openhands-app (Docker), Frontend: agent-canvas - Host: Minisforum X1 Pro (app/orchestration) - Inference: llama-server on a separate GPU box (DGX Spark), connected over LAN - Use case: dev + one other user, multi-agent tasks via natural-language prompts The pattern (happens repeatedly, same sequence each time) 1. Runs smoothly for a while 2. Suddenly gets logged out mid-session 3. Can log back in after some time, but: - Console shows agent errors - Running processes/prompts slow down noticeably - Sometimes a request just times out and shows
5000ms.
4. To recover, I've had to recreate the Docker container — which wipes conversation history and in-progress work that was otherwise stable This has happened 2-3 full cycles now, always the same shape, so I suspect a config issue rather than random flakiness. Specific questions 1. Is the
5000ms
timeout coming from OpenHands itself (a session/request timeout setting), or is it more likely a reverse proxy/gateway timeout in front of it? Where is that value normally configured? 2. Is there a known failure mode where a stalled or slow LLM backend (e.g., a local llama-server on a remote host) causes OpenHands to force a logout/session drop rather than just queuing or retrying? 3. What's the recommended persistence setup so conversations/workspace state survive a container recreation? Right now everything appears to live inside the container, not on a mounted volume — is that the actual issue? 4. Is there a standard way to get OpenHands health/latency into Prometheus/Grafana (request latency, agent error rate, active sessions) so I can catch degradation before it cascades into a full logout, rather than finding out after the fact? 5. Has anyone run OpenHands stably for multiple concurrent users long-term, and if so, what's different about your deployment (persistent volumes, timeout values, reverse proxy config, process supervision) vs. a default docker-compose setup? Happy to share docker-compose.yml, logs, or network diagrams if that helps narrow it down. Trying to get this stable enough, so want to fix the root cause, not patch symptoms.
e
Just to confirm, is the backend an agent-server? One way to debug might be having OpenHands or any other agent analyze logs and see if it can pinpoint the root cause of the delay
w
Suddenly gets logged out mid-session
WDYM logged out?
To recover, I've had to recreate the Docker container
How are you running everything? What commands did you use?
t
The backend used is openhands-app, which is the agent-server. The session times out and disconnects from the conversation panel, and then I see the local backend page. This is how I created the openhands-app backend that connects to the agent-canvas frontend: sudo docker run -d \ --name openhands-app \ --network app-net \ --restart unless-stopped \ -p 18000:18000 \ -p 8000:8000 \ -p 9091:9091 \ -v /mnt/cephfs-mgmt/openhands-workspaces:/home/openhands/workspace \ -v /var/run/docker.sock:/var/run/docker.sock \ -v bca4791bab06fc1b091f713699ea9e76c946ce21537aa4a41f2df3364427e3f2:/projects \ -v openhands-data:/home/openhands/.openhands \ -e AGENT_SERVER_URL=* \ -e GIT_AUTHOR_EMAIL=openhands@newfire.app \ -e GIT_COMMITTER_EMAIL=openhands@newfire.app \ -e NPM_CONFIG_YES=true \ -e CI=true \ -e OH_ENABLE_VNC=true \ -e GIT_AUTHOR_NAME=newfire \ -e GIT_COMMITTER_NAME="OpenHands NewFire" \ -e npm_config_yes=true \ -e COREPACK_ENABLE_DOWNLOAD_PROMPT=0 \ -e GITHUB_TOKEN=g**** \ -e GPG_KEY=****\ -e POETRY_HOME=/usr/local \ -e ACP_NODE_DIR=/opt/acp-node \ -e LC_ALL=C.UTF-8 \ -e LANG=C.UTF-8 \ -e LOG_JSON=true \ -e EDITOR=code \ -e VISUAL=code \ -e GIT_EDITOR="code --wait" \ -e OPENVSCODE_SERVER_ROOT=/openhands/.openvscode-server \ -e NOVNC_WEB=/usr/share/novnc \ -e NOVNC_PORT=8002 \ -e DISPLAY=:1 \ -e VNC_GEOMETRY=1280x800 \ -e CHROME_BIN=/usr/bin/chromium \ -e PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium \ -e CHROMIUM_FLAGS="--no-sandbox --disable-dev-shm-usage --disable-gpu" \ -e LD_LIBRARY_PATH="/usr/lib/aarch64-linux-gnu/usr/lib/usr/lib/x86_64-linux-gnu:" \ -e LLM_TIMEOUT=3600 \ -e OPENHANDS_AGENT_TIMEOUT=7200 \ -e PROMETHEUS_ENABLED=true \ -e PROMETHEUS_PORT=9091 \ -e OTEL_EXPORTER_OTLP_ENDPOINT=jaeger \ -e OTEL_SERVICE_NAME=openhands-agent \ ghcr.io/openhands/agent-canvas:latest
w
which wipes conversation history and in-progress work
You can mount the conversation data to keep your history around!
v openhands-data:/home/openhands/.openhands \
should be handling this, I'm unsure why it's not. Do you have data in
./openhands-data
?
it sounds like there might be a memory leak or something causing the server to slow over time...I wonder if you could check the memory/CPU usage on the container in initial state vs slow state?
t
Yes I do, I have been using different LLM’s to debug so maybe that's a cause because almost everytime I recreated the container some conversations left and others that I thought were not there anymore came back. I will find out how to check the memory and cpu usage now and get back to you now.
w
ohh there's a weird restart bug IIRC--if you end the server and restart it quickly it will leave some conversations orphaned
if you wait ~1m before restarting you should get all your conversations back
t
I see, thanks for the heads up. I was able to get the memory and cpu usage for the openhands-app and agent-canvas which is the sandbox. Everything is under 2% which is not crazy. CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS ac09487c7de2 openhands-app 0.45% 837.4MiB / 91.94GiB 0.89% 1.44MB / 929kB 69.6MB / 266MB 130 84e6c9ba39a2 agent-canvas 1.50% 1.954GiB / 91.94GiB 2.13% 476MB / 6.87GB 159MB / 2.78GB 420
w
are they slow right now?
t
yes it is, i think it might be the model. this is the configuration i have for the qwen 3.6. Description=Qwen3.6-35B LLM Server (port 8090) After=network.target [Service] Type=simple ExecStart=/home/newwave-dgx/llama.cpp/build/bin/llama-server \ --model /home/newwave-dgx/minimax/models/qwen3.6-35b-26gb/Qwen3.6-35B-A3B-RAM-26GB.gguf \ -ngl 99 \ --ctx-size 262144 \ --flash-attn on \ -t 10 \ --host 0.0.0.0 \ --port 8090 \ --parallel 1 \ --jinja \ --alias qwen3.6 \ --no-warmup \ --cache-ram 2048 \ --cache-type-k q4_0 \ --cache-type-v q4_0 \ --batch-size 1024 \ --override-kv general.context_length=int:262144 Restart=always RestartSec=10 WorkingDirectory=/home/newwave-dgx Is it not able to handle concurrent requests? That is what I am thinking. Because I changed the parallel parameter to 2, but that cut the context window, which affected the other tasks I was running, and I got a litellm request error.
w
hmm it's possible--there was an issue about LLM requests locking up the server
t
You think so? Because I have been running three conversations at a time, and it seems like nothing has moved for an hour now. But the logs are showing with a http status code of 200 and that it has received the bash request.
thonk 1
Yh i am still checking to find the root cause and this is what hermes had to say when i told it to debug agent-canvas and find out why the conversations were not responding. Hermes ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ Here's the root cause. I found it in the logs. == ROOT CAUSE == The conversation (80ab030e-e05e-464c-be88-50c22e4c3720) has hit a context window overflow and is stuck in a failure loop. The agent (qwen3.6 model via OpenAI-compatible API) only supports a 131,072 token context window. The conversation has accumulated 181,720 tokens — that's ~50K tokens over the limit. Every time the agent tries to take a step, it hits: openai.BadRequestError: Error code: 400 "request (181720 tokens) exceeds the available context size (131072 tokens)" The agent retries, the conversation runs again, same error. That's why you see "Received bash request" every 5 seconds forever — the frontend is polling, the agent is trying to execute, it fails, and the cycle repeats with no error surface reaching the UI. == WHAT I FOUND IN THE LOGS == Three separate errors, all pointing to the same root cause: 1. Conversation 80ab030e (your current one) — 181,720 tokens > 131,072 limit - First hit: 2026-07-08 102714 - Kept retrying, same error, every attempt fails 2. Conversation 29277710 — 183,981/183,983 tokens > 131,072 limit - Hit at 2026-07-08 102913 and 161906 - Same problem, same model limit 3. Conversation 14f3c5040e — missing agent.critic.api_key config - Validation error, can't even start The "Received bash request" every 5 seconds is just the frontend polling loop. The agent never completes a step — it fails before it even gets to output anything. == THE FIX == You need to either: A) Start a new conversation (this one is beyond recovery — the context is too large to trim from inside the running agent) OR B) If you want to keep that conversation: - Switch to a model with a larger context window (e.g., Claude 200K, Gemini 1M, or qwen3 with 131K won't help — you need one with >182K) - Or truncate/reset the conversation state manually in the conversation directory The model being used is qwen3.6 via an OpenAI-compatible endpoint, capped at 131K tokens. You're over by ~50K. Want me to help you pick a model with a larger context window, or do you want to just start fresh? ---------------------------------------------- The problem with this answer is that it does not correlate with the configuration I sent to you earlier. Found out just now that it was a Litellm context window mapping bug that intercepted the process. Still digging and hopefully this can help.
I found out that the litellm model registry never had the context window correct, which meant it defaulted back to the default setting og 131K, so now I have to ensure that litellm knows the right context size for my model as 256K so it can override the default. Please let me know if I am taking the long way around; it is my first time going this in-depth with development.
w
ahhh yeah that does sound like an issue 😕 sorry this was so hard to track down
t
Yh it was, thanks for the apology. We are to help each other 🙏🏾
highfive 1