Hi, I wanted to ask about the error message:

[ERROR] livekit.agents: process did not exit in time, killing process

I’ve noticed that this error appears in the logs whenever my shutdown hook is not triggered. What could be the reason for this, and how can I determine whether the issue is coming from LiveKit itself or from my implementation? From the logs, it seems like the error is being emitted by LiveKit. Also, what would be the best way to further investigate this and identify what might be causing the process not to shut down gracefully?

Yes, it's coming from LiveKit, specifically here: https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/ipc/supervised_proc.py#L257. Do you have any long running processes or operations in your agent? Looks like something isn't getting closed down properly.

@lively-winter-88668 I don't think any process long enough for the default 30 seconds to timeout. So, basically 1. The shutdown hook is not getting called as i have a console log at the start of the hook and it never gets printed in this case. 2. Most of the process are some DB queries or some API calls to my internal system.

So, the issue happens 20% of the time. And it's only when the issue happens that your shutdown hook is not called? Maybe the call to DeleteRoom is not returning properly even though the room is being successfully deleted. I still suggest taking a look at all your blocking calls to see what might be getting stuck.

Thanks @lively-winter-88668 let me check.

@lively-winter-88668 For reference, I have two sessions. One of them never called the shutdown hook, while the other did. I also noticed that the problematic session never sent the

shutting down job task

nor the

session closed

log. Failing session:

RM_dspLmVgdrDav

Working session:

RM_hFpMCLw5Mqr3

@great-bear-58988 Yes, I thought it should do, because of this. I don't see any difference in the server logs between your failing room and your working room.

@lively-winter-88668 There's a clear difference towards the end, in all the problematic rooms, I don't see those 2 events (i.e

session closed

and

shutting down job task

) Maybe the session never gets closed due to which, the shutdown webhook never gets called

Where are you running your agent process? What type of hardware is it running on? Are you using a hosting service? If so which one and what types of instances is it on?

• Running on self-hosted AWS ECS container using python SDKs • Uses 8 vCPU and 16GiB of memory (3 tasks run at a time with auto-scaling on) • AWS ECS is our primary hosting service

Which processor class are you using also Is this EC2 or Fargate?

Our Launch type is Fargate

I think that is the issue

By default, its 30s and the max I can set is 120s 🤔

120s is not enough to drain an agent in most cases

Just to clarify, are you assuming that a new Fargate task is spawned for every inbound call? Our Fargate task is actually a long running process and does not terminate when a call ends

First we do not recommend using Fargate as there are issues like when you are draining it has a limit that does not allow it to drain. You should use EC2 if you are using ECS.