Hey guys, Is it possible for anyone to help me un...
# agents
r
Hey guys, Is it possible for anyone to help me understand what happend here? It seems the agent exited from the call before the caller did? Not sure how the agent was triggered to leave. https://cloud.livekit.io/projects/p_54i9l8qbrru/sessions/RM_JDT9WoszgzyX
t
It's about how you handle things in Backend @rapid-painting-51133. If you end the session then the agent might exit from the room even though user is present. Either you can listen to events in Front end and remove the user or if you forcefully want the user to remove from the room you can use
delete_room
method
r
I understand that however, these isn't any way for just the agent to leave the room in my code. I am only using delete_room, and I know that didn't get invoked because both the sip participant and egress were still in the room for an extra 2 mins
@lively-winter-88668 do you think you could help me out. Thanks!
l
I can't see anything obvious in the server logs why the agent left, do you have agent logs? You might try enabling observability on your project if this happens again: https://docs.livekit.io/agents/ops/deployment/logs/#view-runtime-logs
r
Yea we do have logs enabled, the traces show that the agent decided to leave in the middle of the conversation. This is the first time I am seeing something like this as well
@bored-accountant-30277 maybe you could help me here. Thanks!
Hey guys, just wanted to follow up. We received a very bad review becuase the agent just stopped talking in the middle of the conversation so I want to find the root cause so we can prevent it from happening again.
l
@rapid-painting-51133 which session ID triggered the negative review? Let's try to get to the bottom of this.
r
This is the sesssion id RM_JDT9WoszgzyX @lively-winter-88668
Please let me know if you need anything else
l
I see a client-initiated leave in the server logs (i.e. the agent left the room, rather than being kicked) and you can see the same in your room events at the bottom of https://cloud.livekit.io/projects/p_54i9l8qbrru/sessions/RM_JDT9WoszgzyX. Do you happen to have agent logs for this? I don't see any recent known issues in this area, but also good advice to be on the latest version of the Agents SDK.
r
Yes I have the agents logs of data dog, what specifically do you need? The logs don't so show anything other then the agent leaving
We were on the latest version as well
r
Yes, logs for that agent session would be helpful to see why the agent is leaving the room. Are you able to reproduce this with a basic agent like basic_agent.py On possible issue is your agent is blocking a thread too long and it appears to be unresponsive. I recommend trying to simplify you agent as much as possible to help isolate why it is leaving the room.
r
I tried reproducing it however I am not getting the same behavior
What are instances where
_self_.ctx.add_shutdown_callback
gets invoked? I believe this is getting invoked prematurley
r
You call that to register your shutdown hook:
Copy code
async def on_shutdown():
        logger.info("Post-conversation activity: webhooks, cleanup, etc.")

ctx.add_shutdown_callback(on_shutdown)
r
right but what is invoking the shutdown hook?
r
are there any logs on your side that could indicate why the agent left the session in the middle of the conversation?
r
Should be in your agent logs
What session?
our agent logs on data dog don't really show us why the agent left
r
Participant left (CLIENT_INITIATED)
r
Right but why is that happening with the agent?
how would it initiate itself leaving the room?
I would expect that with the sip participant
r
I don't know what your code looks like. But there are various way to initiate a leave by the agent
r
We are using delete_room
👍🏼 1
but i can't see why that would only make the agent leave?
is it possible?
I don't think delete_room is what is causing this
The room only ended 6 mins after the agent left
The stream closes and the session ends right after
Is there any reason why the session closes when the agent leaves?
based on these logs both session cosed and shutting down job task logs are getting emitted, but in fact the session was still open for another 5 mins
r
Not really sure what your agent code looks like
r
I am more asking why the logs are like this
at 13:43 the agent left the session, but the logs say the session was closed and the job is shutting down
however if you look above in the lk dashboard we can clearly see the session was live for another 5 mins
r
sounds reasonable. Not sure what the logic you are trying to do does
r
I am just trying to figure our what causes the agent to leave mid session 😂
Do you have any idea of the triggers that allow just the agent to leave without closing the entire session?
r
Yes. Agent can leave and session will not end
r
What function triggers that?
r
r
hmm this is for server initiated actions
mine was initiated by the client
two different things i beleive?
r
ok, I thought you said your agent left and call stayed up
r
yes that is what happend
r
Then not sure why you now say "mine was initiated by the client"
You can
await session.aclose()
to leave room for agent
And leave the call up
r
I think we are misunderstanding each other. If you look at the logs the agent left due to
disconnect_reason: "CLIENT_INITIATED"
, but the session still stayed up
How is it possible the agent leaves due to "CLIENT_INITIATED", if the commands you provided here are server initiated
r
I don't know what your code is doing. Sorry. Not really sure what you are talking about
r
Do you understand my question?
I don't think the code is nesscary
I am more trying to understand how it works
Let me try again step by step.
r
ok. What is your usecase and what are you trying to do?
r
Can we both agree that in the LK dashboard on this screen shot we are seeing the agent leave at
Dec 16, 2025, 3:13:43 PM
and the reason is due to "CLIENT_INITIATED"?
r
I can look. But what are you trying to do?
r
I am trying to figure out why the agent left mid session
r
What do you see in your agent logs?
r
Because the sip participant was on the session talking to himself for 5 mins without any responses causing a complaint to our customer.
In my agent logs I see
Copy code
{
id: "AE_3FEHSgttFvsf"
project_id: "p_54i9l8qbrru"
room_id: "RM_GKiEdYVzvQjT"
timestamp: "2025-12-16T21:13:43Z"
type: "Participant left (CLIENT_INITIATED)"
participant_id: "PA_7QUFEBaxZSdT"
participant_identity: "agent-AJ_p6MvcdkWiMZY"
node_id: "NM_OASHBURN1B_KLLzQXoXKe4T"
timestamp_us: 1765919623741509
disconnect_reason: "CLIENT_INITIATED"
}
r
That is not agent log right? That is server log
r
These are the agent logs, leading up to the shutdown
r
Hmm, so what did that tool execution do?
r
It called our backend with some information, returned a 200 with no errors
Please let me know if that makes sense
r
You don't have anymore logs for
CA_a3VMH4HcX3n6
CAW_4vCobW9YPB5F
?
r
I just have the lk server and agent logs
r
we need the agent logs
Not sure what is going on
r
I've sent above in the screenshot
How would you like them sent?
r
well, that does not really have any of the info in it
r
what exactly are you looking for?
r
The full agent logs
r
The agent logs only give so much
How do I get that? Datadog only gives me so much information
r
there are commands like this
Copy code
lk agent logs
r
okay i am grepping logs
Is there a way to search by room?
or id?
@lively-winter-88668 any way you know of getting the agent logs easier?
@refined-appointment-81829
Is there any way I can get help on identifying the number of times this has happened? Basically a query that tells me how many times the agent has left the session due to "CLIENT_INITIATED"
l
Hi @rapid-painting-51133, I took a look at https://cloud.livekit.io/projects/p_54i9l8qbrru/sessions/RM_WRf5iAn4zX7P (from 15th Dec) and it looks like it shows the same. Reading through this thread, I think the difficulty is there is only so much we can see in the server logs - from our point of view, we see the agent leave but we're not sure why, and the hope is that information is contained within the agent logs. You previously shared agent logs that showed
input stream detached
- it would be good to understand more behind that, but I guess that information was not sent to DD. Do you see any agent logs in Insights? https://docs.livekit.io/deploy/observability/insights/#logs Also, It looks like you have https://docs.livekit.io/deploy/agents/cloud/logs/#datadog enabled, but can you change the log level perhaps?
r
Hey @lively-winter-88668, thank you for your insight. I am having trouble even looking for the logs. My data dog logs don't contain the entire picture, unless I am missing something obvious. The log level is currently set at debug, which I would assume is appropriate.
I've tried looking through the code, but I am not able to pinpoint an exact location where the agent leaves with a client initiated status. Are you familiar with any where this can be happening? Maybe its happening automatically, something that is not being triggered intentionally by me.
https://cloud.livekit.io/projects/p_54i9l8qbrru/sessions/RM_YcZkx3btZ2uA/observability?mode=logs @lively-winter-88668 it happend again, but this time I've enabled the agent insights
Are you seeing anything here that is abnormal?
I still can't really tell why the agent decided to leave
@refined-appointment-81829
r
I don't have access to those logs. Can you reproduce this with basic_agent.py in your dev environment and share the logs from that client when the issue happens?
You can send the download and share the agent insights data too.
r
observability-RM_xWn67DuxgLhq.zip
observability-RM_YcZkx3btZ2uA.zip,observability-RM_f654nZJGZhrw.zip
Three cases where it has happend today
r
Why are those HTTP requests failing?
r
Its due to azure filtering policy, which is a different issue, and we have retries, however don't think its related as the other examples don't have it
Is there any known way of forcing the agent to stay in the session?
Could this be related to some concurrency issue? Where our agents drop if we reach N parallel sessions?
r
That case would be next agent does not answer
r
The agent isn't answering here either
Could it be concurrency limits on the external providers? TTS or STT?
@lively-winter-88668 do you know what causes a job to shutdown? Are there any triggers that happen without an explicit invocation by the developer?