I took some time on defining a better architecture for the streaming component of our software, as our current setup is far from optimal. I've attached an HTML documenting and outlining our existing issues (including six bugs directly linked to the current design) along with proposed solutions for a more solid architecture. The HTML is just minimal to explain the problems we have now, why we should change, and in what we should change. (I have more precise notes locally that I can also share). I would appreciate feedbacks/ideas/critics on the design. cc. @limited-student-10747 @wooden-umbrella-84098 @calm-artist-7206

Only nit I can offer is because I've been in an ACP rabbit hole today. I think you'll want to use messageId, which was added in 0.12.0. You'd probably run into this doing the implementation even if it wasn't mentioned, tho. agentclientprotocol.com/rfds/message-id

Great doc! A quick remark: When I implemented the LLM APIs (without intermediary, no litellm), I noticed that the streaming deltas were recognizable from the start, so they could be replaced by their final form. I don’t recall the exact details, but they all were recognizable - all 4,

v1/completion

,

v1/responses

, Gemini (their old API), and Anthropic. Not sure off-hand why our deltas are apparently not; I wonder if we use or inherit some litellm class that doesn’t…, and the append-only log design

Only nit I can offer is because I've been in an ACP rabbit hole today.

I think you'll want to use messageId, which was added in 0.12.0.

You'd probably run into this doing the implementation even if it wasn't mentioned, tho.

Yes. A problem was also that with ACP we didn't have the

messageId

.

*c*an we do this nicely and safely, considering the append-only

EventLog

? Store it in-memory in Conversation?

This should be safe because minting isn't a write. The

EventLog

never assigns ids, it receives events that already have one.

Do I understand correctly, are we saying that we try to write a durable event, before it’s finished?

So today we announce an event on the socket before it's on disk, and if the append then raises we've already told the client about an event that doesn't exist. The change just swaps them so publish happens only on a successful append.

I wonder a bit if we really need to mask secrets contained inside the LLM response.

I think we must do that 🙂 but I have strong opinions on this theme. Best case scenario, we block the LLM to read secrets when it is using tools. But this is difficult to maintain as we have a lot of tools, right?

One thing that surprised me in Joe’s latest bug report was the number of events: we appear to count all streaming deltas ? It doesn’t seem correct, deltas are fleeting, not Events.

Yes. This is actually a bug. Now the problem is that the deltas are a subclass of events.