https://livekit.io logo
Join Slack
Powered by
i need your help on pod sizing for a LiveKit agent...
# ask-ai
a
i need your help on pod sizing for a LiveKit agents Kubernetes deployment
t
Thanks for reaching out, Sebastián. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I can help with pod sizing for a LiveKit Agents Kubernetes deployment. To give accurate recommendations, could you share a few details? • Expected concurrent agent sessions (range or peak) • Will agents use resource-heavy features (noise cancellation, VAD, long-running STT/LLM/TTS)? • Target latency or performance requirements (soft vs. strict) • Any existing hardware instance types or CPU/memory limits you prefer? • Do you plan to autoscale (HPA) or use fixed replica counts? • Typical session duration (minutes) and traffic pattern (spiky vs. steady) If you want a quick baseline while you gather that: start with 4 CPU / 8 GB per pod (supports ~10–25 concurrent voice sessions) and use CPU-based autoscaling with scale-up threshold ~50% and agent 
load_threshold
= 0.7. Also set a longer terminationGracePeriodSeconds (10+ minutes) to allow sessions to drain. Sources: Self-hosted deployments | LiveKit Documentation | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit, with interactive elements
a
• we have a constant and predictable steady load 24/7. now is around 60 but aim to move to 100, 150,200 in a few hours as we battle test. this agents are mostly short lived (around 5mins) and kicked and then new agents spawn into new rooms immediately • all are S2S agents that use provider server side VAD, no noise cancellation but Realtime pipeline • we use HPA • we are using 8vCPU, 16GiB nodes • we have now setup one pod per node using all available cpu and nodes • load_treshold is set to default 0.7 and gradePeriod is 5minutes
t
Thanks — this is helpful detail. I’ll respond to your current setup and the upcoming scale target (60 → 100/150/200 steady sessions), given 8 vCPU / 16 GiB nodes, one pod per node, HPA, S2S short-lived sessions (~5 min), provider VAD, Realtime pipeline, and 
load_threshold=0.7
with 5 min grade period. Recommendations • Baseline pod sizing ◦ Keep pods at 4 CPU / 8 GiB as the starting unit. This matches LiveKit guidance and real-world tests for voice agents and gives room for headroom and autoscaling responsiveness. ◦ On your 8 vCPU / 16 GiB nodes, run 2 pods per node (each with 4 CPU) rather than one pod using the whole node. This reduces single-point contention and improves bin-packing for short-lived sessions. • Expected capacity per pod ◦ Estimate ~10–25 concurrent short sessions per 4 CPU pod. Because your sessions are short and S2S with server-side VAD (no heavy noise cancellation), use the higher end of that range for planning — assume ~20 sessions/pod for steady-state capacity. • Cluster sizing for targets (rounded) ◦ For 100 concurrent sessions → ~5 pods → with 2 pods/node → 3 nodes (ceil). ◦ For 150 concurrent sessions → ~8 pods → 4 nodes. ◦ For 200 concurrent sessions → ~10 pods → 5 nodes. (These use 20 sessions/pod estimate; if you prefer conservative 10 sessions/pod, double pod/node and node counts.) • Autoscaling settings ◦ Keep 
load_threshold=0.7
on agents. Configure HPA to scale on CPU utilization and trigger scale-up earlier: target CPU ~50% (or 0.5) so new pods are ready before agents hit 0.7. ◦ Use a short scale-up stabilization/aggregation window (e.g., 1–2 minutes) and a longer scale-down cooldown (e.g., 5–10 minutes) to avoid rapid thrashing given short session durations. ◦ Consider using Cluster Autoscaler with node provisioning short delay to match HPA speed. • Worker/availability tuning ◦ Reduce 
gradePeriod
only if you see slow reaction to load changes — with steady 24/7 load and 5-min sessions, your existing 5-minute gradePeriod is reasonable; keep it unless you observe saturation. ◦ Ensure 
terminationGracePeriodSeconds
is >= session max (5–10 minutes) so pods can drain active sessions cleanly. • Operational checks during your ramp ◦ Monitor per-pod CPU and memory, agent 
load_fnc
value, pod restart rates, and rejection/queue metrics. ◦ Run a quick canary scale to 100, then 150, watch for sustained CPU >70% or increased rejections before proceeding. ◦ If you see CPU saturating earlier than expected, switch to a conservative capacity assumption (10 sessions/pod) and add nodes. If you’d like, I can: • Recompute exact node counts using a different sessions-per-pod assumption (10, 15, or 20). • Suggest HPA config YAML snippets for target CPU and stabilization windows. Which follow-up would be most helpful? , with interactive elements
a
right now we have deployed with this one pod per node given resource request:
Copy code
# Long enough drain to let in-flight calls finish through a deploy.
terminationGracePeriodSeconds: 300
# Outbound-WS workers don't need a preStop sleep.
preStopSleepSeconds: 0

# Pin to the dedicated cloud agent pool (e2-custom-8-16384)
nodeSelector:
  workload: livekit-agent-cloud

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "livekit-agent-cloud"
    effect: "NoSchedule"

# QoS shape for an async I/O-bound voice workload on e2-custom-8-16384:
#   - CPU: requests only, no limits (CFS throttling causes audio jitter).
#   - Memory: request == limit (Guaranteed QoS, last to be evicted).
# One pod per node — leaves headroom for kube-system + brief bursts.
resources:
  requests:
    cpu: "7000m"
    memory: "12Gi"
  limits:
    memory: "12Gi"

readinessProbe:
  enabled: true

# maxUnavailable: 1 (instead of minAvailable: N) is self-adjusting as
# minReplicas grows: always exactly one pod can be drained at a time, no
# matter the scale. Avoids the trap where minAvailable: 1 lets all-but-one
# drain in parallel once you have many replicas.
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

autoscaling:
  enabled: true
  # Two warm pods for HA. Bump as you migrate more users to LiveKit Cloud.
  # PDB (maxUnavailable: 1) needs minReplicas >= 2 to allow node drains.
  minReplicas: 2
  # Effectively uncapped. Real ceiling is the cluster's node-pool max + GCP quotas.
  maxReplicas: 1000
  # Below the agent server's load_threshold (0.7). Lower as a hedge while CPU
  # is the only signal — for I/O-bound Realtime workloads, CPU underestimates
  # true load.
  targetCPUUtilizationPercentage: 50
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
t
Thanks — this is a solid config and you’ve thought through many of the right trade-offs. I’ll flag a few targeted tweaks and checks to improve capacity, stability, and autoscale behavior for your steady 24/7 short-session workload. Summary of what I’m addressing: your current 1-pod-per-8vCPU node config with requests cpu=7000m, memory=12Gi, no cpu limits, HPA target 50%, minReplicas=2, terminationGracePeriodSeconds=300, podDisruptionBudget maxUnavailable=1. Recommendations • Run 2 pods per 8vCPU node (each with 4 CPU / 8Gi) instead of one 7 CPU pod. ◦ Benefits: better bin-packing for short-lived sessions, reduces single-point contention, and matches LiveKit guidance (4 CPU / 8 GiB baseline). ◦ Action: change requests to cpu: "4000m", memory: "8Gi" and allow two pods/node via node allocatable headroom. • Keep CPU requests only (no CPU limits). ◦ Reason: avoids CFS throttling which can cause audio jitter. • Memory: keep request == limit for Guaranteed QoS. ◦ Action: set memory limit to "8Gi" to match the request. • HPA / scaling tuning ◦ Keep agent 
load_threshold = 0.7
. ◦ Keep HPA target CPU ~50% (good). This gives headroom before agents mark themselves overloaded. ◦ Use short scale-up stabilization window: 60–120s. ◦ Use longer scale-down stabilization/cooldown: 300–600s to avoid thrash given 5-min sessions. ◦ Ensure HPA metrics reflect node-level allocatable vs pod requests so new pods can schedule. • PodDisruption and draining ◦ Keep terminationGracePeriodSeconds >= session length (300s is okay for ~5 min sessions). If sessions occasionally exceed 5 min, increase to 600s. ◦ PDB maxUnavailable: 1 is good for rolling updates. Ensure minReplicas >= 2 for PDB safety. • Capacity planning (using ~20 sessions / 4 CPU pod estimate) ◦ 100 sessions → ~5 pods → with 2 pods/node → 3 nodes. ◦ 150 sessions → ~8 pods → 4 nodes. ◦ 200 sessions → ~10 pods → 5 nodes. ◦ If you prefer conservative sizing (10 sessions/pod), double pods & nodes. Validate with canary load tests. • Short-lived sessions & HPA interplay ◦ Because sessions are ~5 min and spawn immediately, prefer quicker scale-up and slightly delayed scale-down so pods are available for the next sessions. ◦ Consider maintaining a small buffer of warm pods (minReplicas >= 2–3) above steady-state if you see frequent cold starts. • Operational checks to run during ramp ◦ Track per-pod CPU, memory, agent load_fnc, agent rejection/queue metrics, pod start time, and scheduling latency. ◦ Watch for CPU near 70% before autoscale triggers — if that happens prematurely, drop sessions/pod estimate or increase pods. ◦ Monitor kube-scheduler pending pods to ensure cluster autoscaler is bringing nodes fast enough. If you want, I can: • Produce a revised manifest snippet with requests/limits, HPA stabilization windows, and PDB adjustments for 2 pods/node. • Recalculate nodes/pods using a conservative 10 sessions/pod assumption. Which follow-up would be most useful? , with interactive elements
a
i've updated the config with some missing values, could you re-evaluate
t
Thanks — I can re-evaluate. Please paste the updated deployment manifest (the full pod resources, HPA settings, PDB, and any changed fields like 
terminationGracePeriodSeconds
, 
preStopSleepSeconds
, 
minReplicas
, 
maxReplicas
, 
targetCPUUtilizationPercentage
, and 
gradePeriod
) so I can review exact values and give precise recommendations. , with interactive elements
a
Copy code
# Long enough drain to let in-flight calls finish through a deploy.
terminationGracePeriodSeconds: 300
# Outbound-WS workers don't need a preStop sleep.
preStopSleepSeconds: 0

# Pin to the dedicated cloud agent pool (e2-custom-8-16384)
nodeSelector:
  workload: livekit-agent-cloud

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "livekit-agent-cloud"
    effect: "NoSchedule"

# QoS shape for an async I/O-bound voice workload on e2-custom-8-16384:
#   - CPU: requests only, no limits (CFS throttling causes audio jitter).
#   - Memory: request == limit (Guaranteed QoS, last to be evicted).
# One pod per node — leaves headroom for kube-system + brief bursts.
resources:
  requests:
    cpu: "7000m"
    memory: "12Gi"
  limits:
    memory: "12Gi"

readinessProbe:
  enabled: true

# maxUnavailable: 1 (instead of minAvailable: N) is self-adjusting as
# minReplicas grows: always exactly one pod can be drained at a time, no
# matter the scale. Avoids the trap where minAvailable: 1 lets all-but-one
# drain in parallel once you have many replicas.
podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

autoscaling:
  enabled: true
  # Two warm pods for HA. Bump as you migrate more users to LiveKit Cloud.
  # PDB (maxUnavailable: 1) needs minReplicas >= 2 to allow node drains.
  minReplicas: 2
  # Effectively uncapped. Real ceiling is the cluster's node-pool max + GCP quotas.
  maxReplicas: 1000
  # Below the agent server's load_threshold (0.7). Lower as a hedge while CPU
  # is the only signal — for I/O-bound Realtime workloads, CPU underestimates
  # true load.
  targetCPUUtilizationPercentage: 50
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
t
Thanks — I’ve reviewed the updated manifest and your constraints. This config is solid, but given your steady 24/7 short-lived S2S sessions and 8vCPU/16Gi nodes, I recommend the following targeted changes and checks. Key issues • One pod per node with cpu request 7000m wastes bin-packing on short 5‑minute sessions and increases single‑point contention. • No CPU limits is correct to avoid CFS jitter, but memory limit should match request for Guaranteed QoS. • HPA scaleUp stabilizationWindowSeconds: 0 is very aggressive for short-lived sessions; scaleDown window 600s is appropriate. Recommended changes (minimal, actionable) 1. Run 2 pods per node: ◦ Set resources.requests.cpu: "4000m" and requests.memory: "8Gi" ◦ Set limits.memory: "8Gi" ◦ This yields 2 pods/node on your 8vCPU/16Gi nodes and matches LiveKit 4 CPU / 8 GiB baseline. 1. Adjust HPA behavior: ◦ Keep targetCPUUtilizationPercentage: 50. ◦ Set scaleUp.stabilizationWindowSeconds: 60–120 (instead of 0) to avoid aggressive flapping while still responding quickly. ◦ Keep scaleDown.stabilizationWindowSeconds: 300–600 (600 is OK). 1. Pod disruption / drain settings: ◦ terminationGracePeriodSeconds: 300 is fine for ~5 min sessions. Increase to 600 only if sessions sometimes exceed 5 minutes. ◦ Keep preStopSleepSeconds: 0 if you rely on graceful termination only. 1. Min replicas / warm buffer: ◦ Keep minReplicas >= 2. Consider minReplicas = 3 if you see many immediate back-to-back short sessions to reduce cold-starts. 1. Capacity assumptions & node counts (using 20 sessions / 4 CPU pod estimate): ◦ 100 sessions → ~5 pods → ~3 nodes (2 pods/node). ◦ 150 sessions → ~8 pods → ~4 nodes. ◦ 200 sessions → ~10 pods → ~5 nodes. ◦ If you want conservative planning, assume 10 sessions/pod and double pods/nodes. Operational checks during ramp • Monitor per-pod CPU, memory, load_fnc, rejection/queue metrics, scheduling latency, and pod start times. • Watch for sustained pod CPU >70% or agent rejections; if seen, reduce sessions/pod estimate or add nodes. • Ensure Cluster Autoscaler can provision nodes within your HPA scale-up timeframe. If you want, I can: • Produce the revised manifest snippet with the 4 CPU/8Gi resources and adjusted HPA windows. • Recalculate node counts using a different sessions-per-pod assumption (10/15/20). Which would you like next? , with interactive elements
a
there isn't an overhead in k8s nodes so we have a bit less than 8cpu and 16gb to allocate?
t
You're correct — Kubernetes reserves some node capacity for system components, so you won't have the full 8 vCPU / 16 GiB available for pods. Quick guidance and actions: • What reduces allocatable capacity: ◦ kube-reserved (kubelet + kube-proxy etc.) ◦ system-reserved (OS, container runtime) ◦ eviction thresholds / kubelet reserved for eviction (node allocatable) ◦ any DaemonSet pods (e.g., logging/monitoring) consuming CPU/memory • Typical reserve values (examples, not prescriptive): ◦ For an 8 vCPU / 16 GiB instance you may see ~0.2–1 CPU and ~0.5–2 GiB reserved depending on your distro and DaemonSets — allocatable commonly ends up around 7–7.5 CPU and 14–15 GiB or less. • Actionable changes for your config: a. Target 2 pods/node with each pod requesting 
cpu: "4000m", memory: "8Gi"
only if your node allocatable >= 8 CPU and >=16 GiB. Because allocatable is lower, confirm allocatable before applying. b. Query actual allocatable on a node: • Run 
kubectl get node <node-name> -o jsonpath='{.status.allocatable}'
(or 
kubectl describe node <node-name>
). • Check current DaemonSet usage with 
kubectl top node
and 
kubectl get pods -n kube-system -o wide
. a. If allocatable is lower than needed for two 4 CPU/8 Gi pods: • Option A: reduce pod requests slightly (e.g., 
3900m
CPU / 
7.5Gi
) only after validating LiveKit performance under load. • Option B: keep pod requests at 4 CPU / 8 Gi and run 1 pod/node (conservative, simpler). • Option C: add a small node-level buffer by setting requests to 3.5–3.8 CPU if tests show capacity per pod remains acceptable. a. Confirm Cluster Autoscaler and HPA consider allocatable resources so new pods can schedule. • Recommended immediate checks: ◦ Run 
kubectl describe node
to see Allocatable CPU/Memory. ◦ Check kubelet flags or cluster config for 
kube-reserved
and 
system-reserved
values. ◦ Inspect DaemonSets using node resources (fluentd/metrics). If you paste the output of 
kubectl describe node <one-of-your-nodes>
(Allocatable section) I’ll recompute whether 2x 4CPU/8Gi pods per node is safe or suggest exact pod requests to fit your real allocatable capacity. , with interactive elements
a
allocatable values are 7.91 cpu and 13 GiB, i would give a bit of headroom to this, 12GiB is fine for memory but CPU likely 7.5 so there's 0.4 cpu for daemon sets
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!
2 Views
Open in Slack
PreviousNext
Powered by