I am getting 503 error when tested using an agent for egress recording and not able to receive the recording in s3 for the entire number of calls, with 4 cpu and 4 gb only few calls out of 10 or 20 is fetched

Thanks for reaching out, Rima. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

so at the agent side when we are getting 503 twirp error, what i observe at egress workload is - connecting to redis. my egress is in private.

Thanks — that’s a helpful observation. You mentioned the agent sees 503 Twirp errors and the egress workload logs show “connecting to redis,” and that your egress service is in a private network. I’ll walk through the most likely networking/Redis causes and then ask for a few specific diagnostics so I can narrow this down. Quick summary of what I’m addressing • Why the agent receives 503 Twirp errors during egress. • The egress logs show attempts at “connecting to redis.” • Your egress service is deployed in a private network. Likely causes • Redis unreachable from egress workers (networking, security groups, or routing blocking access), causing egress workers to fail processing requests and return 503. • DNS resolution issues for the Redis host inside the private network. • Redis auth/credentials or TLS mismatch causing connection retries (logs often show “connecting to redis” if auth fails and the client keeps retrying). • Redis cluster/node not accepting connections or over capacity. • Egress workers attempting to reach a Redis endpoint that’s only accessible from a different VPC/subnet (e.g., accessible from public agents but not from private egress). • Firewall, NACLs, or corporate proxy in the private network blocking outbound connections or Twirp RPC responses. Targeted checks to run now 1. From an egress worker host, test basic connectivity to Redis: ◦ TCP test:

telnet <redis-host> <redis-port>

or

nc -vz <redis-host> <redis-port>

◦ If you use TLS, test with

openssl s_client -connect <redis-host>:<port>

◦ Note any DNS resolution failures or timeouts. 1. Validate Redis host & port the egress process is configured to use: ◦ Confirm the environment variables or config contain the correct Redis hostname, port, and whether TLS is required. 1. Check Redis authentication/TLS: ◦ Verify the password/ACL or TLS certificates are correct. ◦ Look for authentication errors in egress logs (e.g., “ERR invalid password” or TLS handshake failures). 1. Review network path and permissions: ◦ If redis is in another VPC/subnet, confirm route tables, VPC peering, Transit Gateway, or PrivateLink are configured. ◦ Check security groups and NACLs allow inbound traffic from the egress worker subnets to the Redis port. ◦ If your egress is private and the Redis host uses a public DNS name, ensure DNS resolves to the private IP inside the VPC (split-horizon DNS). 1. Inspect Redis health and capacity: ◦ Ensure Redis is up, not overloaded, and accepting new connections. Look at connection counts and slow logs. 1. Increase egress worker logging (short term) ◦ If not already verbose, enable more detailed egress logs around Redis connection attempts so we can see specific error messages. What I need from you to pinpoint the issue • A sample of the egress worker log lines around the “connecting to redis” messages and the 503 Twirp error timestamps (remove any secrets). • The Redis connection string/config used by egress (host, port, whether TLS/auth is enabled — do not include the password). • How Redis is deployed (managed Redis like AWS Elasticache, self-hosted in another VPC, etc.). • Whether egress and Redis are in the same VPC/subnet or different VPCs, and whether there are NAT/proxy/VPC peering/PrivateLink in between. • The output of a connectivity test from an egress host (e.g., result of

nc -vz <redis-host> <port>

or

telnet

) if you can run it. Quick remediation suggestions you can try immediately • If Redis is unreachable due to networking, route/security changes to allow connectivity will stop the 503s. • If using burstable or private instances with strict outbound rules, temporarily allow outbound access to Redis from the egress subnet and test. • If auth/TLS mismatch is suspected, confirm config matches Redis requirements and test with a simple Redis client from the egress host. • If Redis is overloaded, scale Redis or add more egress worker capacity so requests aren’t rejected. Next step Share the logs and the connection details listed above (no secrets). With those I’ll analyze the precise Redis error and give exact remediation steps (network config rules, config fixes, or scaling recommendations).

Hi @proud-policeman-65107, I recently faced a similar issue https://community.livekit.io/t/egress-service-gets-stuck/298 did you check the egrees logs? is it getting stuck on "connecting to redis" and nothing thereafter? a single egress job requires 4 cores of compute, so if your egress service is running on a 4 core machine, it would only be able to process only one concurrent job. You can validate this by switching on the debug logs.

Thanks — that context is very helpful. I’m addressing your question about whether egress can get stuck on “connecting to redis,” and the steps to confirm and fix it. What I’m responding to • You see 503 Twirp errors at the agent, egress logs show “connecting to redis,” egress runs in a private network, and you may be running on a 4‑core machine (one egress job needs ~4 cores). Short answer • Yes — an egress job can appear stuck at “connecting to redis” and return 503s if Redis isn’t reachable or if workers are overloaded/throttled. Also, a 4‑core host can typically run only one Room Composite egress at a time; concurrent jobs will queue or be rejected. Why this happens (concise) • Networking/Redis reachability: If the private egress cannot reach Redis (DNS, routing, SGs, NACLs, peering, PrivateLink), the worker will keep retrying to connect; requests may time out and return 503 Twirp errors. • Redis auth/TLS or capacity: Wrong credentials/TLS or an overloaded Redis can cause repeated connect attempts and failures. • Resource constraints: Room Composite uses headless Chrome and is CPU intensive; a single job can consume ~4 cores. On burstable instances, CPU credit exhaustion can also cause timeouts even when average CPU looks fine. • Worker queueing: Egress workers accept requests based on current load; if all workers are busy or blocked retrying Redis, new requests may be rejected with 503. Immediate checks to run (ordered by fast wins) 1. Turn on debug logs for egress and reproduce the error so logs capture the full connect attempt sequence. 2. From an egress host, test connectivity to Redis: ◦ TCP: nc -vz <redis-host> <redis-port> (or telnet) ◦ TLS: openssl s_client -connect <redis-host>:<port> (if TLS enabled) ◦ Note DNS resolution, timeouts, or handshake errors. 1. Confirm Redis config used by egress: ◦ Hostname, port, TLS=true/false, whether auth is enabled (do not share secrets). 1. Check that egress and Redis are routable: ◦ Same VPC/subnet? If different, confirm VPC peering/PrivateLink/route tables and security groups/NACLs allow traffic. ◦ If using a public DNS name from a private VPC, verify it resolves to the private IP (split-horizon DNS). 1. Inspect egress worker logs around the “connecting to redis” lines for specific errors (auth failure, TLS error, connection refused, timeout). 2. Validate instance type / CPU behavior: ◦ Are you on a burstable instance (t3/t4)? Check CPUCreditBalance. ◦ If on a 4‑core machine running Room Composite jobs, try reducing concurrency or use a larger instance. What to share so I can pinpoint the root cause • A snippet of egress logs around the “connecting to redis” messages and the 503 Twirp times (redact secrets). • The Redis connection config used by egress (host, port, TLS enabled? auth? — no passwords). • The result of a connectivity test from an egress host (copy/paste output of nc/telnet/openssl). • Your instance type (e.g., t3.large, m5.large) and whether egress is in the same VPC as Redis. • Whether you’re running Room Composite or Track Egress (Room Composite is the more CPU-heavy case). Short-term mitigations you can try now • If Redis is unreachable: update routing/security rules to allow egress hosts to connect to Redis. • If Redis auth/TLS mismatch: verify egress config matches Redis requirements and test with a simple client from the egress host. • If CPU bound: reduce concurrent egress jobs, or move egress to larger fixed‑performance instances (m5/c5 family) or add more egress worker instances. • If using Room Composite and you don’t need full layout: try Track Egress for less CPU-intensive tests. If you share the log snippet and connectivity test results, I’ll analyze them and give an exact next step (network config rule, config change, or capacity recommendation) to stop the 503s and get recordings reliably uploaded.

Hey @strong-hydrogen-34992 - Thanks a lot for reaching out back, sorry I missed your message. Yeh so the team was using RoomComposite egress which required more compute - 4 cores as you mentioned - so we were trying to load test using the same :0. Now we switched to Track egress, which takes comparatively less compute. Also there is also a parameter called cpu_cost - that allocates cost for each job for egress. So we tweaked around that param as well.

Thanks — great update, and that switch makes sense. What I’m responding to • You moved from Room Composite (CPU-heavy, ~4 cores per job) to Track Egress (lower CPU) to reduce load and avoid the 503/Twirp errors tied to worker saturation. Quick confirmation • Yes — Track Egress is significantly less compute‑intensive than Room Composite and should reduce CPU contention and the likelihood of workers getting stuck while retrying Redis or spinning up headless Chrome. That change is a correct step for improving concurrency on a 4‑core host or when you want more concurrent jobs per host. What to monitor next (short actionable checklist) • Enable debug/verbose logs on egress and watch for any remaining “connecting to redis” or Twirp 503 messages. • Monitor CPU and memory on your egress hosts during load tests to confirm headroom. If you use burstable instances, also monitor CPU credit balance. • Watch Redis connection metrics (accepted connections, connection errors, latency) to ensure no intermittent auth/TLS or network issues remain. • Track egress webhooks (

egress_started

,

egress_ended

) to verify jobs finish and uploads succeed. If upload failures occur, check

egress_ended

for

backupStorageUsed

and

manifestLocation

. If you still see 503s after switching • Share a short log snippet around the 503 and any “connecting to redis” lines (redact secrets). • Share the egress type shown in the request (Track Egress) and your instance type. • Run a simple connectivity test from an egress host to Redis and paste the result (nc/telnet/openssl). Next step • Try another controlled load test with Track Egress while watching the above metrics. If everything looks good, the switch resolved it. If not, send the logs/connectivity output and I’ll pinpoint the remaining issue and recommended fix., with interactive elements