https://docs.all-hands.dev/ logo
Join Slack
Powered by
Sorry for the vague bug, but LLM Profiles are in a...
# proj-agent-canvas
w
Sorry for the vague bug, but LLM Profiles are in a bad state. Just saw 3 different people fail to do initial onboarding due to token issues https://github.com/OpenHands/agent-canvas/issues/1146 CC @purple-thailand-64770 @ancient-car-4311 this will be a tricky one to track down. I might have openhands investigate and suggest how to refactor/simplify how we're managing LLM Profiles. Could be both in agent-canvas or agent-server
nod 1
p
We were just talking about this (cc: @straight-sundown-84319 and @flaky-jordan-67349). I'm noticing issues in Agent Canvas and SaaS myself.
a
@modern-fireman-99755 had also been looking into LLM profile issues so may be good to coordinate with him as well.
➕ 1
p
Yeah, that was the rough outcome of our discussion - see what Vasco comes up with and go from there.
nod 1
👍 1
Still worth having OH take a pass too, though, I think!
m
Looking into it...
ty 1
c
Vasco was just fixing the behavior on delete!
m
I found the problem... Opening a PR for that in a moment. Let me validated
❤️ 4
But the underlying problem is that LLM profiles are at the core of a lot of components in the agent-canvas, and they have a lot of corner cases. What I noticed is that it is very easy to make a change that seems small but ends up breaking the profiles. I'm not an expert in TypeScript testing, but could we implement integration tests (with real API calls, similar to what we do in the SDK) that reproduce LLM profile workflows so that we can be sure we aren't breaking anything? We could just have them run before a release and whenever a profile-related PR is opened. @quick-window-91705 are our E2E tests heading in this direction? I think this is a must-have to ensure stable releases in the future.
Vasco was just fixing the behavior on delete!
@calm-artist-7206 this is coming but I need a release of the sdk before 🙂
➕ 1
q
https://github.com/OpenHands/agent-canvas/pull/1140 - adding coverage here https://github.com/OpenHands/agent-canvas/pull/946 - more coverage here, which discovered some of the corner cases initially
ty 1
❤️ 2
🎉 1
c
I set off my agent to investigate how much are we still using non-profile llm settings; I think that can be a significant cause of error prone redundancies/codepaths
m
@ancient-car-4311 @wooden-umbrella-84098 @ancient-car-4311 A small summary about all the bugs in the issue: 1. First time setup is often plugging in OpenHands as the provider This should be fixed in the PR I just opened, please double check if you can reproduce the bug or not. PR: https://github.com/OpenHands/agent-canvas/pull/1148 2. Deleting the active LLM profile doesn't set the next one as active, leaving you in a weird state where convos don't have an LLM For that I need a release of the SDK, which should come in the next hours. So tomorrow I can fix that. 3. LLM profiles randomly switch when starting a conversation This was fixed this morning in this PR: https://github.com/OpenHands/agent-canvas/pull/1123
w
Thanks for the investigation! I would be in favor of anything we can do to simplify/refactor to make this more stable, even if we have to sacrifice things like backward compatibility
👍 1
c
Yes, we’re just looking into another cause of error-prone special cases: 
openhands/
provider. We persist it “translated” with the litellm_proxy base URL, which leads to all client code (cloud, cli, canvas) writing the same code to recognize it was 
openhands/
😓 and ‘translating’ it back in various spots I think actually we only need to ‘translate’ it just before we send to litellm, who indeed doesn’t know it. It’s a provider for all the rest of the code 🤔 Keep openhands/* as public LLM model; translate to litellm_proxy only at transport time
❤️ 1
👍 1
2 Views
Open in Slack
PreviousNext
Powered by