https://docs.all-hands.dev/ logo
Join Slack
Powered by
The agent doesn't seem to use finish tool reliably...
# proj-agent
q
The agent doesn't seem to use finish tool reliably, it requires explicit user prompting I wonder if the system prompt or tool prompt can be made more assertive
c
What LLM was it?
q
gpt 5.5
c
We should not rely on Finish tool, can we avoid it? The agent-server has an endpoint to get the last agent message, which returns the message either from Finish or from a plain text message
Is this problematic for Automation?
q
Yeah structured output is super useful and is supported through the finish tool docs.openhands.dev/sdk/guides/structured-output#…
nod 1
m
Which automations are you speaking about?
You can change the system prompt of the automation to "force" the LLM to use it.
c
I kinda said that in the wrong order, my thinking is that yes, maybe we can nudge some LLMs more (Sonnets/Opus don’t seem to need a nudge), but we still shouldn’t be sure Aha… What if we added a hook to tell the LLM on plain message “if you finished use Finish tool” or something like that?
👀 1
m
However, it is always random... So if you can find another way it is much better
q
I tested a simple echo hello world automation and other ones like summarize todays news and its not reliably using unless I specifically add it as part of the automation prompt "use finish tool before you wrap up" but if i include the tool, i want that behavior to be automatic instead of re-prompting
so prompting does work, but i have to do it explicitly myself
m
Can you not just extract the final msg if the LLM doesn't use the tool?
q
i want the agent to specify the structured output metadata (my custom fields), im not interested in the final msg per say
c
What are your thoughts on a mechanism like: • (maybe) have hooks in python, or a simple yaml for this problem, it’s a common thing • in automation definition, be able to reference a hook • e.g. maybe it can be as simple as 
hooks: structured_finish
q
cool i'd like to see if that can help, but id also like to tweak the tool prompt a bit maybe bc it would be easier solve if that worked out
c
My thinking is that… not all Automations want this… Some do, but something like Slack events doesn’t, it should be fine IMHO to just use the text there; though I’m in two minds if we’re doing the right thing here mixing so deeply messaging bridges with “real” automations (those without human in the loop)
You're welcome to try prompting, but of course if you need to rely on it… umm 😇
m
You're welcome to try prompting, but of course if you need to rely on it… umm 😇
Agree with that... If you can do that in an heuristic way is much better... also hook can help but it will never give you 100% of what you want.
What are the info you want to extract?
q
any custom structured outputs fields on the finish tool its just that the agent doesn't use the tool, so there is no information to extract
m
ok but what are the info that we need? What is the user case?
q
in my case, whether the agent thinks its blocked or not
or a pr link metadata
depends on the automation
m
Another way is to prompt the agent to finish with a specifi format and then you can take the last msg and extract that. This should work better then the finish tool as LLM are trained to do that (because of benchmarks).
but as always: you will not have 100% correctness
c
The hook can check if the agent respected the format
and reject and ask it again
👍 1
it can be a deterministic check (or not, but this in case deterministic would work )
q
yeah i feel that's kinda what structured output is for, to make it more reliable than not and have consistent output schema i think using a hook to see whether finish tool was used sgtm
👍 1
github.com/OpenHands/automation/pull/405 adding finish tool hook check to the automation preset
✅ 1
seems to work well
m
ah ok cool... but this is just for prompt-based automation right?
7 Views
Open in Slack
PreviousNext
Powered by