I have been spending quite some time debugging the chat context which exists in an agent activity (single reply), an agent and in the session (shared across agents). I have been exploring the code and the docs, but there doesn't seem to be a single resource that indicates when what is synced and how pushing to one manually impacts behaviour or even multiple handles to impact different elements. Is there a resource I am missing here?

would this works?

const ctx = session.currentAgent.chatCtx.copy();
ctx.insert(newItem);
await session.currentAgent.updateChatCtx(ctx);

I'll try this out! My main question remains: what is the relation between the chat contexts and how/when and why are they synced?

I think the session one is for logging and reporting purpose (i.e this also inform the livekit frontend ui to see this) and the agent chat context is the one that LLM actaully use and see it does not look like there is sync job between them and you might have to update them both like it is done elsewhere in the code @bitter-ghost-46918 can confirm on this

The code snippet @quaint-ambulance-32873 shown above should be the correct way to update chat history for an agent! Regarding difference,

agentSession.history

is the global

chatCtx

. It keep track of all chat items across all agents during the session lifetime (as well as for analytic purpose).

agent.chatCtx

only keep track of chat history of that active agent

I tried this approach too, but still it's not part of any of the contexts. I attached an image of the actual code I'm trying to run. The values of the opts.ctx.session.chatCtx are updated and I get the most recent values from it, but pushing to it does not seem to have any effect. I have checked both the agent chat context and the session chat context. Not sure what additional information could be helpfull?

but pushing to it does not seem to have any effect

You should always call

agent.updateChatCtx

to update the chat context of a agent. Inserting into chat ctx object itself won't do anything.

As I am adding this back into context without the exact same tool name, I am violating this behaviour.

If you have a different tool set then chat context

copy

will automatically filter out all tool calls & outputs that are not part of the given tool set. This is required for models such as openai & gemini & anthropic, where it does not allow unrecognized tool name to appears in the chat history. I'm curious what specific feature you'd like to implement. Maybe I can help you to find another way

So in a multi agent chat behaviour I would first have an agent that has knowledge on how to find things through specific tools after which it would hand off to a different agent that acts upon that data with a different set of tools. Additionally in our case we do parallel processing which has a different tool name. The agent itself can call

delegate_tool_call

which has a param of

create_note

. This tool responds immediately with a success message to keep the conversation flowing. After the parallel execution I would like to report back to my agent by inserting the

functionCall

and

functionCallOutput

of the

create_note

toolcall so that it could follow up if it went wrong or use it in subsequent conversation. This feature is blocked by the filtering

The agent itself can call

delegate_tool_call

which has a param of

create_note

. This tool responds immediately with a success message to keep the conversation flowing. After the parallel execution I would like to report back to my agent by inserting the

functionCall

and

functionCallOutput

of the

create_note

toolcall so that it could follow up if it went wrong or use it in subsequent conversation. This feature is blocked by the filtering

I'm not entirely understand the approach here.. What's the intention behind calling a tool (delegate_tool_call) that calls another tool (create_note), what prevents you from just making

create_note

as a tool? And where does the parallel execution comes in? We do support parallel tool call for openai, which might be what you are looking for, or if I'm missing something

The

create_note

tool call is one that can take a lot of time so we don't want to block the response of the agent. There are several of these tools that I want to run on the side. I could just name that delegation to

create_note

but then after the actual execution I still want to provide confirmation or errors within the conversation. If I modify the original tool call messages, the agent history assumes that it knew this when generating the replies so it doesn't mention anything about it. Secondly if I hand over to an agent that does not have the

create_note

then this message will still be filtered out right? Should I create summaries when handing over between agents?

Yep, I see what you mean. Happy to hop on a call. I'm free for the next 3 hours

Amazing I also noticed the updated docs for the background audio during calls. We'll try to get that in this week!