Thanks — I’m responding to your question about diagnosing the root cause of slow API responses when starting egress composite at ~40 concurrent users. Summary diagnosis • CPU/memory being ~30%/50% suggests raw host resources aren’t the primary limit. • The likely bottlenecks are orchestration/queueing and per-egress initialization overhead, not simple CPU saturation. Specifically: ◦ Egress worker pool saturation / queued requests (workers accept based on load). ◦ Chrome + GStreamer startup and per-recording process overhead (init latency, process limits). ◦ Redis Pub/Sub or queue backlog slowing request dispatch. ◦ I/O, network, or process-count limits (disk/network throughput, file descriptors). ◦ Per-request latency compounding at scale (each await blocks API flow). Practical diagnostics to run next 1. Check egress queue and worker utilization ◦ Measure queue depth / pending egress requests in Redis. ◦ Observe number of active egress workers vs. queued requests. 1. Instrument per-request timing ◦ Record timestamps for: API create-room → dispatch agent → start egress request → egress accepted → first media frame captured. ◦ Identify which step adds the most latency. 1. Inspect egress host/process details ◦ List Chrome/headless and GStreamer processes per egress and their CPU/thread/FD usage. ◦ Check process counts, open file limits, and ulimits on the egress server. 1. Check I/O and network metrics ◦ Monitor disk write throughput/latency and network bandwidth/packets for egress servers. ◦ Check for spikes or saturation during peaks. 1. Review egress logs ◦ Look for Chrome/GStreamer startup delays, retries, or worker reject messages. ◦ Look for Redis timeouts or connection errors. 1. Test controlled load experiments ◦ Run a small-scale test increasing concurrency while collecting the above metrics to see where latency jumps. ◦ Compare behavior with and without pre-warmed egress (see next item). Short-term mitigations to validate • Pre-warm: Pre-initialize a small number of egress workers/Chrome instances so per-request startup is reduced. • Increase worker pool / scale out egress servers: Add egress instances to reduce queueing (try this in a controlled test). • Tune load threshold: If self-hosted, consider adjusting `load_threshold`/`load_fnc` so workers accept more requests (careful — validate limits). • Offload I/O: If recording to disk is I/O-bound, try writing to faster disks or using remote object storage async buffers. If you want, tell me: • how you currently dispatch egress (agent config, any custom load_fnc), • whether you see queued requests in Redis, and • one sample timing trace for a slow request; I’ll help interpret those results and recommend targeted next steps (pre-warm sizes, scaling targets, or config changes). Sources: Universal Egress | LiveKit | Universal Egress | LiveKit | Server options | LiveKit Documentation | LiveKit Community Day | LiveKit, with interactive elements