Can I get some feedback on the API compliance moni...
# proj-agent
w
Can I get some feedback on the API compliance monitor: https://github.com/OpenHands/software-agent-sdk/pull/2204 Should limit the issue Robert calls out above until we can get the root cause identified
c
Please correct me if I make the wrong assumption, but could we do this on the
view
, rather than on
state._events
?
What is not in the
view
doesn't get to the LLM anyway
w
That's true, but the
view
properties assume the event list is well-formed by the time the view is made, that way they can just focus on maintaining those properties instead of enforcing them. That way the views don't need all the events every time they're made/updated.
It also feels natural that we'd want to keep the event log in as workable a state as possible. It's the original source of truth,
view
objects are just views.
c
You may want to see this too • https://github.com/OpenHands/software-agent-sdk/pull/2256 Also, IMHO we could maybe consider to merge your PR on this • https://github.com/OpenHands/software-agent-sdk/pull/2141 As you can see, Xingyao is trying something interesting and problematic: to somehow fix the duplicate tool result. But the reason for that appears to be that it made it into the event log already.
I said problematic because an earlier iteration of that PR had a
transform
method on events; and because it creates a new
Event
which has to be in the right order. Maybe it's the right thing - because it's necessary, but it's interesting that it needs to. I don't know if we can assume that the event log is workable
cc: @chilly-france-44439
w
I see how that PR started, too: trying to capture the duplicate tool result idea as a
ViewProperty
. Unfortunately those properties can only be used to define contiguous groups (you have to limit their expressivity or the condenser has an impossible job), and the failure case might have the
AgentErrorEvent
pretty far from the actual tool call observation. So I'm not surprised that the transform in the
events_to_messages
function is where Xingyao landed.
We'll probably need some kind of in-situ fix to get old conversations with that weird event situation working again. But we should also do what we can to ensure it doesn't happen in the first place. The API compliance monitor is a real heavy-handed way to do this, but until we can address the upstream issues it should help.
By upstream issues I mean: we've got a common situation where the runtime acts up while a tool call is going and we end up with the
AgentErrorEvent
and the actual observation. Maybe we should work to ensure that error event is only added if the tool is actually broken?
c
I'm a bit unsure on the idea that it will never happen in the first place. I mean, it's an event log in the SDK in a single place, so it feels like we could fix it completely... on the other hand, we're playing with weird timing issues, and in V0 I recall we had a similar problem: something like, execution doesn't return for too long, then it does, but in the meantime we had to give the LLM something so it's an AgentError, so now it exists as an Event but the Obs returns in a split second I mean, OK, if I put it this way it feels fixable, I just have some feeling that we might be better off to accept weird things happen
I could be wrong
w
I think we're on the same page. We'll never eliminate the weirdness, but we'll be well-served by making the agent code as predicable and reliable as possible.