Hello everyone, I've been self-hosting LiveKit and running into issues at scale, specifically with egress composite recording. I have an API that creates a room, dispatches an agent, and initializes egress composite. If any of these steps fail, the API raises an error. At scale (~40 concurrent users), the API slows down significantly mainly due to waiting on egress. CPU usage on the egress server peaks at ~30% and memory at ~50%, so resources don't seem to be the bottleneck. What I'm trying to understand: what is the actual bottleneck here? Autoscaling or pre-warming a few egress servers would likely help, but I want to diagnose root cause first. I've considered skipping the await on egress initialization to make the API respond faster, but the issue is that audio recording is required for my use case, so I can't just fire-and-forget and move on. Any guidance appreciated!

how do you know the API slow down is due to waiting on egress

Hey @flat-photographer-63184, I had tried to move egress to background. So whenever API is called, it would trigger egress init, and instead of await return early along with room details. But that just moved the slow down from create, to background. Infact multiple calls failed due to livekit engine failed, egress didn't start,etc. But I was looking for a way to do this before returning api response, because if egress doesn't start, that call doesn't really matter to us much.

The reason I ask is egress doesn't block, although it can take some time for it to find a slot across the cluster, and could be slow if the cluster is very big

Yes, it doesn't by default, but im awaiting for it. Reason for this is, if egress failed due to any reason, we want to fail early. Instead of completing the whole call without egress.