Why is the initial LiveKit caller connect to Agent...
# ask-air
red-army-65692
12/19/2025, 4:12 AMWhy is the initial LiveKit caller connect to Agent spiking CPU usage significantly. Also why is the initial caller connect to agent time consuming and not instantaneous?
t
tall-airline-16756
12/19/2025, 4:12 AMThanks for reaching out, Sreekanth. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
tall-airline-16756
12/19/2025, 4:13 AMThanks — I’m responding to your two concerns: the CPU spike when the initial caller connects to an Agent, and why that initial connect isn’t instantaneous.
Why CPU spikes on the first caller connect
• The Agent is treated as a full room participant, so connecting triggers multiple CPU‑heavy tasks at once: WebRTC signaling/ICE work, media encode/decode, and any real‑time ML work (STT, VAD, LLM inference orchestration, TTS).
• Agent implementations often load models or spin up inference pipelines on demand; loading or warming these models causes short CPU bursts.
• If the agent process was idle or “cold” it may perform startup work (cold start) when the first call arrives, concentrating work into that initial period.
• Using burstable VM types (e.g., t3/t4) can expose CPU throttling or credit issues that make bursts worse.
Why the initial caller->agent connect can be slow
• WebRTC connection setup (certificate exchange, ICE candidate gathering, DTLS/SRTP setup) takes non‑trivial time.
• Cold starts for the agent or external services (STT/TTS/LLM) add 10–20s delays when the agent must start or warm up.
• Network proximity and downstream service latency (LLM/STT/TTS endpoints, TURN servers) increase round‑trip times and push overall connect/response time up.
• If microphone capture or other client‑side preparations are deferred until after signaling completes, perceived connect time increases.
Practical mitigations you can apply now
• Pre‑warm the agent process and models at startup so inference pipelines and VAD/STT models are loaded before the first caller.
• Use non‑burstable instance types (e.g., m5/c5) for Agents to avoid CPU credit throttling.
• Enable “instant connect” (capture microphone input before full agent join) to reduce perceived latency.
• Use preemptive generation (start preparing responses before user finishes speaking) to improve conversational responsiveness.
• Deploy agents closer to your users and AI service endpoints to reduce network RTT; ensure TURN/TURN detection is configured appropriately.
If you’d like, tell me:
• whether your Agents currently cold‑start (shut down when idle) and what instance types you run on, and
• whether you’re using instant connect or preemptive generation—
and I’ll recommend the most effective next steps for your setup.
Sources: Quotas and limits | LiveKit docs | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Simple Call Answering Agent | LiveKit docs | Agent speech and audio | LiveKit docs
r
red-army-65692
12/19/2025, 4:16 AMThe AWS ES2 that hosts LiveKit and agent is T3. Medium
t
tall-airline-16756
12/19/2025, 4:16 AMThanks — that helps. Running LiveKit + Agent on a t3.medium explains a lot. T3 instances are burstable and can run into CPU credit limits and throttling during the heavy, concurrent work that happens on first connect (model loads, WebRTC/ICE, STT/TTS, encryption). That causes both the CPU spike and longer connect times.
Recommended next steps
• Move to a non‑burstable instance (recommended: m5 or c5 family) to avoid CPU credit throttling and get consistent performance.
• Pre‑warm the agent and models at startup so VAD/STT/TTS/LLM components are loaded before the first call.
• Enable instant connect so microphone capture begins before the full agent join to reduce perceived latency.
• If possible, scale out: run agent processes on a separate instance(s) from LiveKit so media/signaling and inference work don’t contend for the same CPU.
• Monitor CPU credits and CPU steal/IO on the t3.medium in the short term to confirm throttling (CloudWatch metrics: CPUCreditBalance, CPUSurplusCreditsCharged).
If you want, I can:
• Recommend a specific instance size (based on expected concurrent calls) and a quick checklist for migrating agents off t3, or
• Outline the exact CloudWatch metrics to watch and thresholds that indicate throttling. Which would you prefer?
r
red-army-65692
12/19/2025, 4:17 AMWhat if I use c6g.16xlarge
t
tall-airline-16756
12/19/2025, 4:18 AMThanks — I’m responding to your question about using a c6g.16xlarge for LiveKit + Agent.
Short answer
• c6g.16xlarge is an excellent choice. It provides 64 vCPUs and 128 GiB RAM, giving large, consistent CPU capacity (no burst credits) which addresses the initial CPU spikes and connection latency caused by model loads, WebRTC/ICE work, STT/TTS/LLM processing, and cold starts.
Why this helps
• No CPU credit throttling — consistent performance for heavy, parallel tasks.
• Plenty of vCPUs — simultaneous media processing, inference, and signaling won’t contend as much for CPU.
• Large memory — room for preloaded models and buffers to stay resident, reducing cold‑start work.
Practical recommendations when moving to c6g.16xlarge
• Pre‑warm agent processes and load VAD/STT/TTS models at startup so the first call doesn’t trigger heavy loads.
• Run LiveKit and agent processes on separate instances or containers if you want strict isolation; a single c6g.16xlarge can host both but separating improves fault isolation.
• Validate your software on ARM64 (Graviton2) — compile/test native binaries and container images for ARM.
• Monitor CPU usage, latency, and packet rates after migration (CloudWatch or your APM) to size concurrency and tune thread/process counts.
• Keep TURN and AI service endpoints geographically close to reduce RTT.
If you want, I can:
• Recommend a concurrency estimate (instance sizing guidance) if you tell me expected concurrent calls and whether you’ll run LiveKit + agent together or separately, or
• Provide a quick checklist for ARM/Graviton compatibility and a migration plan. Which would you like? Sources: Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Going beyond a single-core | Deploying LiveKit | LiveKit docs
r
red-army-65692
12/19/2025, 4:19 AMIs RHEL better than ubuntu for some reason?
t
tall-airline-16756
12/19/2025, 4:19 AMThanks — you’re asking whether RHEL is better than Ubuntu for running LiveKit + Agents. Short answer: neither OS is inherently better for LiveKit; choose based on your priorities.
Key points
• LiveKit and Agents are typically deployed in containers or as self‑hosted binaries, so the OS matters less than container/runtime compatibility and kernel/networking support.
• Ubuntu/Debian‑based images are common in our Dockerfiles, but those can run on RHEL hosts via container runtimes.
• RHEL strengths: enterprise support, long‑term stability, certified ecosystem, stricter security defaults (good if your org requires vendor support or compliance).
• Ubuntu strengths: broader community packages, faster access to newer packages, common in cloud images and dev workflows (may simplify testing/builds).
• Performance differences for LiveKit itself are negligible; bottlenecks are more often instance type (CPU/memory), network, and container configuration.
If you tell me which matters most (enterprise support/compliance, ease of use, package freshness, or existing infra standard), I’ll recommend the best fit and any migration checklist (container/runtime checks, kernel/network tuning, or ARM/Graviton compatibility if using c6g). Which priority should I base the recommendation on?
Sources: Deploy to a VM | LiveKit docs | Deploy to Kubernetes | LiveKit docs | Builds and Dockerfiles | LiveKit Docs
r
red-army-65692
12/19/2025, 4:21 AMHow to "Pre‑warm agent processes and load VAD/STT/TTS models at startup so the first call doesn’t trigger heavy loads."?
t
tall-airline-16756
12/19/2025, 4:21 AMI’m responding to your request for how to pre‑warm agent processes and load VAD/STT/TTS models at startup so the first call doesn’t trigger heavy loads.
Short steps and example
1. Add a per‑process setup function (prewarm) that runs once when each worker process starts. Load heavy models there and store them on the process userdata.
2. In your session entrypoint, reuse the preloaded models from
proc.userdata Copy code
from livekit.agents import AgentServer, JobProcess, JobContext
from livekit.plugins import silero, inference
def prewarm(proc: JobProcess):
# load VAD into process memory once
proc.userdata["vad"] = silero.VAD.load()
# initialize STT/TTS/LLM clients if needed
proc.userdata["stt_client"] = inference.STTClient(api_key="...") # example
proc.userdata["tts_client"] = inference.TTSClient(api_key="...")
server = AgentServer()
server.setup_fnc = prewarm
@server.rtc_session()
async def entrypoint(ctx: JobContext):
vad = ctx.proc.userdata["vad"]
stt = ctx.proc.userdata["stt_client"]
tts = ctx.proc.userdata["tts_client"]
session = AgentSession(
vad=vad,
stt=stt,
tts=tts,
preemptive_generation=True,
)
await session.start(agent=MyAgent(), room=ctx.room)
await ctx.connect()r
red-army-65692
12/19/2025, 4:23 AMWe see that recording is the one which is taking quite a bit of cpu
t
tall-airline-16756
12/19/2025, 4:23 AMThanks — I’m responding to your note that recording is consuming a lot of CPU. Below are focused explanations and actions you can take to reduce recording CPU usage.
Why recording uses high CPU
• Room Composite uses headless Chromium + GStreamer for layout rendering and encoding; each composite job can consume multiple CPU cores.
• Transcoding (video resizing, re-encoding) is CPU‑heavy; higher resolution/framerate = much more CPU.
• Running egress (recording) on the same instance as agents causes contention with inference/real‑time tasks.
• Burstable instances (t3/t4) throttle under sustained encoding, worsening performance.
Immediate mitigations
• Move egress to dedicated instances (non‑burstable, compute‑optimized like c5/c6). Don’t run egress on agent hosts.
• Use Track Egress or audio‑only Room Composite when possible instead of full Room Composite.
• Lower recording resolution/bitrate (use 720p or audio_only).
• If you must composite video, limit concurrent Room Composite jobs per host (plan ~2–4 cores per composite job).
• Enable autoscaling for egress workers (scale by
livekit_egress_availabler
red-army-65692
12/19/2025, 4:27 AMrecommend instances for audio-only agent. Assume 25 concurrent users. HA is needed as it will be inbound calling.
t
tall-airline-16756
12/19/2025, 4:27 AMThanks — I’m responding with instance and HA recommendations for an audio‑only LiveKit agent supporting 25 concurrent users.
Recommended sizing
• Instance size per agent node: 4 vCPU / 8 GB RAM (e.g., AWS c6a.xlarge or c5.xlarge). This size handles ~10–25 concurrent audio jobs with headroom.
• Instance count for HA: minimum 2 instances across different AZs; recommended 3 for safer failover and rolling updates.
• Worker count: run multiple agent worker processes per instance (default worker pool), targeting ~5–10 concurrent sessions per worker process.
Architecture & operational guidance
• Use non‑burstable compute (c5/c6/m5 families) to avoid CPU credit throttling.
• Separate responsibilities: run agents on dedicated instances; do not host egress/recording on the same machines.
• Autoscaling: enable autoscale (Kubernetes HPA or managed autoscaler) based on CPU or custom agent load metrics; scale up at ~50% CPU to avoid saturation.
• Graceful shutdown: set terminationGracePeriodSeconds ≥ 600s so inbound calls finish during deploys.
• Spread instances across AZs and use health checks so jobs fail over automatically if a worker dies.
Networking & latency
• Place instances near telephony/AI endpoints (TURN, STT/TTS) to minimize RTT for inbound calls.
• Monitor CloudWatch/Prometheus for CPU, latency, and agent session metrics.
If you want, I can:
• Estimate exact instance count for different concurrency targets (e.g., 25 vs 50 vs 100 concurrent users), or
• Provide a Kubernetes manifest/HPA example and termination settings for this setup.
Which follow‑up would you like?
Sources: Deploying to custom environments | LiveKit docs | Agents Framework | livekit/livekit_composite | DeepWiki | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | examples/drive-thru | livekit/livekit_composite | DeepWiki | OpenAI and LiveKit partner to turn Advanced Voice into an API
3 Views