https://docs.all-hands.dev/ logo
Join Slack
Powered by
Hey, Two (relative small) PRs: - <https://github....
# proj-agent-canvas
m
Hey, Two (relative small) PRs: • feat: support LLM profiles on cloud backends • fix: stream LLM tokens for cloud conversations and profile switches (otherwise you have streaming in agent-canvas locally but not when you use cloud as BE)
👀 1
❤️ 1
m
Hello @modern-fireman-99755, I've left comments on both pull requests for your review. Thank you very much! thankyou
b
The lack of multiple LLM profiles was stopping me from using agent-canvas
BTW, I noticed that when you remove a profile from the old OH frontend, the agent-server continues using that deleted profile for the condenser or for the /btw messages
and I also noticed an issue with propagating the LLM timeout settings to the condenser LLM
it's like agent-server continues using the LLM settings that were available when the conversation was created
m
> The lack of multiple LLM profiles was stopping me from using agent-canvas Do you mean: agent-canvas + cloud as BE? About the other msgs: did you have already open issues for them?
Hey @millions-afternoon-47582 • 1 PR: I think I fixed but still I cannot test as I'm an owner in the Org, i.e., I have right to change LLM profiles. Moreover, I cannot put myself just as a member, and test the changes • 2 PR: I added a video
m
1 PR: I think I fixed but still I cannot test as I'm an owner in the Org, i.e., I have right to change LLM profiles. Moreover, I cannot put myself just as a member, and test the changes
I’ve added you as a member of the organization.
ty 1
m
I will test it and re-ping you once is good ;)
thankyou 1
@millions-afternoon-47582 It seems to work. I attached a video in the PR. Please let me know if there is something other that I should fix 😉
thankyou 1
m
I’ve left a comment on the first pull request and approved the second one. Thank you very much! thankyou
m
Hey @millions-afternoon-47582 I follow your suggestion. To do that, I had to open another PR in the OpenHands repo: https://github.com/OpenHands/OpenHands/pull/15048 Moreover, I discover (just randomly) another problem. I opened an issue, and will fix this in another PR (for reference: https://github.com/OpenHands/agent-canvas/issues/1553)
thankyou 1
✅ 1
b
Do you mean: agent-canvas + cloud as BE?
In my agent-canvas version, I cannot setup multiple LLM profiles, but I can select the profiles imported from the OpenHands Backend.
About the other msgs: did you have already open issues for them?
No (tings are changing fast and most of the issues are ignored). But I can confirm that changing the LLM profile does propagate changes to agent-server, but not for the condenser LLM and not for the ask_agent LLM. To reproduce, select another LLM profile and see if it is really used when sending a 
/btw
m
@nutritious-balloon-65098 check if the above msg is really a bug and if it is the case open an issue in agent-canvas
n
I'm on it! Vasco can track my progress here.
Confirmed: the 
/btw
part is a real bug path. What I found: • Agent Canvas switches the main conversation LLM via 
switch_llm
with the selected profile config. • 
/btw
calls 
ask_agent
without passing any LLM/profile override. • In the pinned agent-server behavior, 
ask_agent
caches a separate LLM under the fixed usage id 
ask-agent-llm
. If 
/btw
was used before switching profiles, that cached LLM can keep using the old profile after the main conversation switches. • The condenser part looks less clear/currently partly mitigated: the pinned agent-server appears to update the default 
LLMSummarizingCondenser
when it was cloned from the previous agent LLM, but I included it as a regression-test concern. I also checked for duplicates and didn’t find an existing matching agent-canvas issue. Opened: github.com/OpenHands/agent-canvas/issues/1568
b
Copy code
diff --git a/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py b/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py
index 83901ce3..e70762aa 100644
--- a/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py
+++ b/openhands-sdk/openhands/sdk/conversation/impl/local_conversation.py
@@ -1801,17 +1801,17 @@ class LocalConversation(BaseConversation):
self.state.events, additional_messages=[user_message]
)

-        # Get or create the specialized ask-agent LLM
-        try:
-            question_llm = self.llm_registry.get("ask-agent-llm")
-        except KeyError:
-            question_llm = self.agent.llm.model_copy(
-                update={
-                    "usage_id": "ask-agent-llm",
-                },
-                deep=True,
-            )
-            self.llm_registry.add(question_llm)
+        # Always derive from the agent's current LLM so that profile
+        # switches (switch_llm) take effect on subsequent /btw messages.
+        # We intentionally skip the registry cache: caching was
+        # first-write-wins, so the copy was frozen at conversation start
+        # and stale after any profile change.
+        question_llm = self.agent.llm.model_copy(
+            update={
+                "usage_id": "ask-agent-llm",
+            },
+            deep=True,
+        )

# Pass agent tools so LLM can understand tool_calls in conversation history
response = make_llm_completion(
Tested and works, but uses a different approach (see comments)
something similar happens with the LLM used by the condenser
m
I will just go for this solution
In agent-server 
switch_llm()
, refresh/remove the cached 
ask-agent-llm
when the active conversation LLM changes.
Such that you don't have to copy everytime the LLM when you use 
/btw
. I think it is also a smaller change Do you want to open a PR in the sdk?
✅ 1
b
"refresh/remove the cached..." sound good
please open the PR
m
@millions-afternoon-47582 do you think we are good with the PR for agent-canvas and BE profiles?
m
Could you please share the pull request with me? Thank you very much! thankyou
m
m
Have you implemented server-side authorization when calling the LLM profile APIs? From what I can see, it looks like the current PR may be handling authorization on the client side only—please correct me if I’m mistaken. Thank you very much! thankyou
m
Yes, server-side auth is implemented. Profile CRUD (create/edit/rename/delete/activate) now routes through the org-gated endpoint 
/api/organizations/{orgId}/profiles
, which enforces 
EDIT_ORG_SETTINGS
on the app-server. Therefore, a Member gets a 403 even calling the API directly, not just a hidden button. The client (use-can-manage-llm-profiles) reads the same permission from 
/organizations/{orgId}/me
, so client + server agree. This last endpoint was added in the PR: https://github.com/OpenHands/OpenHands/pull/15048 (yesterday)
m
I have approved the pull request. Thank you very much! thankyou
m
Thank you 🙂
b
Hi @modern-fireman-99755, I found another issue with latest agent-server version: When restarting the agent-server, the LLM timeout is set to a value that doesn't exist in any of the LLM profiles. It seems to be some value that was configured in the past (maybe some deleted LLM profile), but I'm not sure where it comes from. Switching the profile back and forth, set's the timeout again until the next agent-server restart.
m
i will look into it. Do you have logs? Or something in this direction?
b
## Issue agent-server is setting wrong timeouts when it starts. ## To reproduce - Stop the agent-server (or stop the container) - Open the conversation in agent-canvas or old OH frontend It will start the container and connect to the agent-server. Sometimes it says "Failed to connect to server", so you will have to repeat (stop again and reconnect). - Once OH succesfully connected (not before), check LLM timeouts inside the container: cat /workspace/conversations/SESSION/*.json | jq . | grep '\"timeout\"'" You will notice that agent-server sets wrong timeouts when it starts. If you switch the LLM profile, it sets the correct values.
I think I found the issue
it may be an issue with an older version of OH
m
are you sure? Because I found something which was corrected in ACP but not for the OH profiles.
b
no, false alarm
issue is still there
m
Did you saw my PR?
b
yes
on the sdk right?
m
yes yes
I could reproduce what you were saying and the PR should fix this behaviour
b
yes, I'm running latest agent-server
I'm still seeing the issue with latest agent-server
👍 1
I'm not running latest OH version
let me check
m
but this is internal, i.e., agent-server... so if you are using the latest image is good
b
ok
Have you tried reproducing the issue in your env?
m
yes
b
does it set a different value when the agent-server starts?
m
yes
b
but this is another issue related with the /btw messages
the issue I'm seeing now is that the agent-server changes the agent-llm timeout during start
m
yes I solved this
We are speaking here because you started the discussion of this problem on this thread instead to open a new one 🙂
b
yes, to keep context
m
I have pre-fix cache between thread. Don't worry 😉
😂 2
b
I'm still not 100% sure if it happens with latest version
Please try this in your updated env (with your PR): - Change LLM profile timeout to 1234 - Restart the container and reconnect OH to the agent-server - Once connected, check if it still uses the timeout of 1234 In my env. (with your agent-server PR), the timeout resets to a previous value.
I believe my app-server is sending a different timeout when it connects
if you cannot reproduce, then it must be my app-server version
Actually, I think maybe agent-server is loading from JSON files when it starts.
Reproduced with latest versions
2 Views
Open in Slack
PreviousNext
Powered by