Thanks — this is a great, detailed report. I’m responding to the event-loop lag spikes you see in EKS that coincide with the Transcribe reconnects (~16s) and cause “flush audio emitter due to slow audio generation” audio cutoffs. Summary of what’s happening (what I’m responding to) • The same image/code works fine locally but on EKS you get systematic reconnects logged as “restarting transcribe session” every ~16s. • Right before/during each reconnect the asyncio event loop shows blocking of 0.5–3.5s (loop.slow_callback_duration), which causes the audio flush warnings and audible cutoffs. • You’ve ruled out credentials, CPU throttling, and UDP/NAT issues. Key difference is network path (VPC DNS / NAT / VPC endpoint) vs local cached DNS. Likely cause (short) • The reconnect path is performing blocking network operations (synchronous DNS resolution, TCP connect, or TLS handshake) on the asyncio event loop in the EKS environment. These blocking operations are much slower in EKS (VPC DNS, NAT/VPC endpoint path) than on your local machine, so each reconnect stalls the loop and causes audio flushes. Why this fits your symptoms • Reconnects every ~16s => an idle/stream timeout triggers reconnection. • The lag builds just before and peaks during reconnect => connection setup (DNS/TCP/TLS) is the blocking work. • Works locally but not in EKS => environment-level networking (DNS resolution or network path) differs. Actionable next steps — quick tests (fastest to try) 1. Pre-resolve DNS and pin the IP in the pod (quick experiment) ◦ On pod startup, resolve the Transcribe endpoint and write it to /etc/hosts so reconnects don’t trigger DNS lookups: ▪︎ Example: ▪︎ IP=$(getent hosts transcribestreaming.ap-southeast-2.amazonaws.com | awk '{print $1}') ▪︎ echo "$IP transcribestreaming.ap-southeast-2.amazonaws.com" >> /etc/hosts ◦ If the spike disappears, DNS resolution was the blocker. 1. Force an explicit endpoint URL (if plugin supports it) ◦ If

livekit-plugins-aws.STT

supports an

endpoint_url

or similar, set it to the regional transcribe URL to avoid extra resolution logic or use an internal VPC endpoint hostname. ◦ If the plugin doesn’t expose this, you can try patching/wrapping the plugin to inject the endpoint into the underlying AWS client. 1. Capture network timings (tcpdump) across a reconnect ◦ Run tcpdump in the pod during a reconnect and inspect timing for: ▪︎ DNS query/response, ▪︎ TCP SYN → SYN/ACK, ▪︎ TLS ClientHello → ServerHello. ◦ Long delays in any of these indicate where the block occurs. 1. Verify if you’re using an AWS VPC endpoint for Transcribe ◦ If you are not using a VPC endpoint, try using one (if available) so traffic stays inside AWS and avoids NAT/IGW. ◦ VPC endpoints often reduce latency and remove external DNS/NAT variability. Mitigations if DNS/TLS is confirmed • Short term: Pre-resolve/pin IP in /etc/hosts (described above). • Medium term: Configure a VPC endpoint for Transcribe or ensure node DNS caching (e.g., CoreDNS caching) is tuned. • Longer term: If the blocking call is inside the Transcribe client on the main asyncio loop, change the client usage to offload blocking operations to a thread pool or use an async-aware HTTP/2 client that doesn’t block the loop. This requires code changes or a plugin update. Info I need to help further (please provide any that you can) 1. Can you confirm whether

livekit-plugins-aws.STT

accepts an

endpoint_url

or allows you to pass a precreated HTTP session/client? If yes, I can give exact code to inject a pre-warmed session. 2. Are you using an AWS VPC endpoint for Transcribe or are requests going out via NAT to the public endpoint? 3. Can you run a tcpdump during one reconnect and share these timings (SYN/TLS handshake timing and DNS timing)? Even a short summary like “DNS response took X ms; TLS handshake took Y ms” will help pinpoint the blocker. 4. Do your EKS nodes have any sidecars (Istio/AppMesh) or network policies that could be affecting connection setup? If you want, we can proceed in this order: 1) Try the /etc/hosts pre-resolve test (fast and non-invasive). Tell me whether the event-loop spikes stop. 2) If not fixed, capture and share simple tcpdump timing details and whether a VPC endpoint exists. I’ll help interpret the tcpdump/DNS timings and propose the safest next code or infra change. Sources: Amazon Transcribe plugin guide | LiveKit Documentation