https://docs.all-hands.dev/ logo
Join Slack
Powered by
Just out of curiosity, what do you guys do when yo...
# general
b
Just out of curiosity, what do you guys do when you reach the maximum context length of a conversation? Doesn't it become a dead end, causing you to lose the conversation unless you do some nasty tricks under the hood? Web UI doesn't even support `/condense`to solve it. (I reduce the condenser 
Max Size
setting and delete the last events to force the condenser and recover the conversation).
Copy code
ERROR:
litellm.BadRequestError: OpenAIException - request (138229 tokens) exceeds the available context size (133376 tokens), try increasing it
c
I use 1000 max_size or so, and the condenser does it job automatically
✅ 1
Whether it reaches first 1000, or it reaches first context window limit, it just runs, summarizes, drops earliest events, and it continues normally
Why would you lose the conversation?
That error… doesn’t look good. Could you please make an issue on it? And tell the LLM and provider too
OpenHands sdk is supposed to recognize context window errors, no matter in what form they come from litellm. That error looks like one! It’s a bug that we don’t recognize it, so that we run the condenser automatically, IMHO cc @wide-doctor-37830
b
yeah, the problem is we are detecting "the request exceeds the available context size",
but llama-server throws: request (138229 tokens) exceeds the available context size (133376 tokens)
seems like they added the token size and now it's not being detected
c
🫠
b
I will just add "exceeds the available context size"
thanks!
np 1
c
Do you want me to send the agent on it, or would you like to make a PR? Maybe with the “(“ and “)” it might be nice… just in case
b
ok, you can ask the agent
c
@nutritious-balloon-65098 Open a PR with the llama.cpp fix, nondraft, and /iterate
n
I'm on it! can track my progress here.
OpenHands encountered an error: **Failed to fetch final response from agent server: HTTP 503 error: no available server ** See the conversation for more information.
c
Error, but it did. 😅
l
Hey @better-monitor-61654, I did encounter this problem. To fix it, I changed the condenser to token-based condensation, then set the condensation length below the max context length, so for 64k context I set it somewhat low, like 56k.
Also, what version are you using? The latest one? I think that some of my fixes that I did to token counting might make this a little better
2 Views
Open in Slack
PreviousNext
Powered by