https://livekit.io logo
Join Slack
Powered by
I'm using KEDA ScaledObject to autoscale the LiveK...
# ask-ai
b
I'm using KEDA ScaledObject to autoscale the LiveKit Egress (https://github.com/livekit/egress) deployment on EKS (v1.11.0) and I'm seeing the fleet pin at maxReplicaCount forever once a burst pushes it there — it never scales back down even when there are zero active recordings. I followed the maintainer's guidance from livekit/egress#1110 (https://github.com/livekit/egress/issues/1110) (livekit_egress_can_accept_request is the correct metric — the docs' livekit_egress_available is broken). My ScaledObject: spec: minReplicaCount: 3 maxReplicaCount: 30 triggers: - type: prometheus metadata: metricType: AverageValue threshold: "1" query: sum(livekit_egress_requests{type="room_composite"}) - sum(livekit_egress_is_terminating) - type: prometheus metadata: metricType: AverageValue threshold: "1" query: sum(livekit_egress_can_accept_request) Observed: after a burst peaked at 30 pods, the fleet stayed at 30 for 6+ hours with the HPA reporting 0/1 (avg), 1/1 (avg) — both triggers "at target." What I think is happening (would love confirmation): livekit_egress_can_accept_request is a per-pod gauge that returns 1 when the pod has admission headroom, 0 when saturated. With all 30 pods idle, every pod reports 1 → sum = 30. KEDA AverageValue math gives: desired_replicas = ceil(sum / threshold) = ceil(30 / 1) = 30 Because sum scales linearly with replicas when no pod is saturated, the trigger always reports desired = current — it never asks HPA to scale down. And KEDA composes multiple triggers with max(desired), so the primary trigger's correct desired = 0 → min=3 is overruled by the secondary's desired = 30. The HPA never reaches its scaleDown algorithm at all. Questions 1. Is my analysis correct? Is a KEDA AverageValue trigger on sum(per_pod_gauge) fundamentally a self-feeding floor when the gauge is replica-proportional in the steady-state direction you want to scale toward? 2. Is there a correct way to use a per-pod 0/1 saturation gauge in HPA? Value type has the same problem (target=3 means HPA tries to drive sum→3, but sum tracks replicas). I can't find an HPA math configuration where this metric drives both scale-up AND scale-down monotonically. 3. For LiveKit Egress specifically — is anyone running a Prometheus-metric-based HPA that works in both directions? I surveyed 4 public real-world deployments (QumulusTechnology/livekit-demo, corezoid/sim-meet, Ajinkya-A3/Livekit-Helm, workadventure) — all use targetCPUUtilizationPercentage (chart default) or no HPA. Zero use the custom Prometheus path. Is the Prometheus-metric autoscaling pattern in the docs aspirational and not actually used? For now I've fallen back to the chart's targetCPUUtilizationPercentage: 60 and that drains correctly, but I'd like to understand whether (a) the KEDA recipe can be made to work, or (b) the per-pod-saturation-gauge pattern is just incompatible with how K8s HPA computes desired replicas. Environment: EKS 1.30, KEDA 2.19, LiveKit Egress v1.11.0 (chart 1.8.4), audio-only RoomComposite workload, audio_room_composite_cpu_cost: 1.0, pod cpu requests=4 / limits=7 (admission gate reads cgroup cpu.max = 7, so each pod admits 5 concurrent recordings).
t
Thanks for reaching out, Dinesh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’ve read your description and the ScaledObject you’re using. Quick acknowledgement: I’m responding to your three questions about whether the KEDA AverageValue trigger on the per‑pod gauge is causing the “stuck at max” behavior, and what correct approaches exist. Short answers • Your analysis is correct: using KEDA’s AverageValue (with a query that sums a per‑pod 0/1 gauge) creates a replica‑proportional metric in idle state, so HPA desired = ceil(sum/threshold) will equal current replicas and prevent scale‑down. • A per‑pod 0/1 saturation gauge exposed as a sum/average directly to HPA/KEDA is fundamentally problematic for scale‑down when the metric scales with replica count. • Workarounds: convert the per‑pod gauge into a cluster‑level metric that reflects actual load (not proportional to replica count), or use different metric types/queries that measure active work instead of admission headroom. CPU-based HPA works as a reliable fallback. Follow-up diagnostic questions (I need one or more to give a precise recipe) 1. Can you share the raw Prometheus samples (or 
sum(...) by ()
values over time) for both 
livekit_egress_can_accept_request
and 
livekit_egress_requests{type="room_composite"}
from before, during, and after the burst? Patterns will confirm whether the sum scales linearly with replicas when idle. 2. Do you control the exporter/query layer (can you add recording‑active metrics or run recording‑count queries)? 3. Are you open to changing the KEDA trigger type to use a Prometheus query that returns a per‑cluster count (e.g., active recordings) or a ratio (active/total pods) rather than sum of per‑pod headroom? Suggested approaches you can try now • Use active-work queries: drive scaling from a metric that counts active recordings (e.g., sum(active_recordings)) so desired_replicas = ceil(active_recordings / capacity_per_pod). This is monotonic both up and down. • Convert to a ratio: compute accepted_fraction = sum(can_accept_request==1) / count(pods); use AverageValue on that ratio or use 
metricType: AverageValue
with a query that returns desired_replicas directly (e.g., ceil(sum(active_requests) / capacity_per_pod)). KEDA can also accept a query that returns a scalar representing desired replicas (depending on KEDA/Prometheus setup). • Use external/advanced queries: create a Prometheus recording rule that emits a cluster‑level 
livekit_egress_cluster_available_slots
= sum(can_accept_request) - desired_floor and then use that to drive scaling (so it does not grow with pods when idle). • Keep CPU HPA for scale‑down: as you have, targetCPUUtilizationPercentage reliably drains because CPU is already per‑pod utilization rather than additive headroom. If you want, share one Prometheus time series excerpt (question 1) and I’ll provide the exact PromQL to: • Produce a cluster active recordings count, and • A KEDA trigger/query that yields desired replicas (so HPA can both scale up and down). Would you like me to craft those exact PromQL/KEDA ScaledObject snippets? , with interactive elements
b
Re: follow-up diagnostics — confirming the self-feeding floor hypothesis Thanks. Below is what I have, captured from the broken-KEDA window before I fell back to chart CPU HPA. 1. Raw Prometheus samples — confirms sum tracks replicas linearly when idle 12 h range queries against our LiveKit-ns Prometheus, step=15 min. (Local Prom retention only goes back ~3 h, so I lose the 13:00 UTC burst itself — but the idle steady-state and the post-fix drain are clean.) ┌─────────────────────────────────────────────┬─────────────────────────┬───────────────────────────────────────┬─────────────────────────────────────────────┬─────────────────┐ │ Time (UTC) │ sum(can_accept_request) │ count(can_accept_request) (=replica │ sum(requests{type=room_composite}) (active │ sum / count │ │ │ │ count reporting) │ recordings) │ (per-pod avg) │ ├─────────────────────────────────────────────┼─────────────────────────┼───────────────────────────────────────┼─────────────────────────────────────────────┼─────────────────┤ │ 16:02 → 18:32 (idle, KEDA active) │ 30 │ 30 │ 0 (with one 1-rec blip at 17:02) │ 1.00 the whole │ │ │ │ │ │ time │ ├─────────────────────────────────────────────┼─────────────────────────┼───────────────────────────────────────┼─────────────────────────────────────────────┼─────────────────┤ │ 18:47 (I deleted the KEDA ScaledObject, │ 7 │ 7 │ 0 │ 1.00 │ │ fleet drained to manually-pinned 7) │ │ │ │ │ ├─────────────────────────────────────────────┼─────────────────────────┼───────────────────────────────────────┼─────────────────────────────────────────────┼─────────────────┤ │ 19:02 (chart CPU HPA had taken over, │ 3 │ 3 │ 1 │ 1.00 │ │ drained to min=3) │ │ │ │ │ └─────────────────────────────────────────────┴─────────────────────────┴───────────────────────────────────────┴─────────────────────────────────────────────┴─────────────────┘ Pattern: sum(can_accept_request) == count(can_accept_request) at every single observation while no pod was at admission cap. Replicas went 30 → 7 → 3 and sum followed identically. The ratio (which is what KEDA AverageValue / threshold=1 computes as HPA "current value") was a flat 1.00 throughout — so the trigger reported "at target" no matter how many idle pods were running. That's the self-feeding floor, captured as data. livekit_egress_requests{type="room_composite"} reads 0 essentially the whole window (with a single 1-recording blip at 17:02 and another at 19:02). So the primary trigger correctly said desired=0, but KEDA max(triggers) was overruled by the secondary. 2. Do I control the exporter/query layer? Partially: - Prometheus / KEDA / HPA wiring — yes, fully under our control (self-hosted). - Egress's exported metrics — no, that's livekit/egress (https://github.com/livekit/egress) upstream. I can't add a new metric inside egress without a fork. - Application-side recording metrics — yes, separately. Our pipeline service writes a call_logs row when a recording starts/stops and we run an SQL-→-Prom exporter elsewhere, so I could expose revrag_active_recordings as a fleet-wide gauge. But that's a workaround that requires us to teach our own service about something egress already knows internally. 3. Open to changing the KEDA trigger type / query shape? Yes, completely open. The fix I shipped (chart targetCPUUtilizationPercentage: 60) drained correctly but lags admission by 30–60 s. I'd rather have an admission-coupled signal. Specific shapes I'd love your opinion on: (a) Per-cluster count using existing egress metrics: sum(livekit_egress_requests{type="room_composite"}) - sum(livekit_egress_is_terminating) This is replica-independent — it's a true fleet-wide active-recording count. As Value trigger with threshold = per_pod_admission_cap (in my case 5), desired_replicas = ceil(active / 5). Drains to 0 → clamps to min when idle. The only "custom" thing is deriving the per-pod-cap from cgroup × MaxCpuUtilization / cpu_cost which feels brittle. (b) Ratio / saturation: 1 - (sum(livekit_egress_can_accept_request) / count(livekit_egress_can_accept_request)) This is a 0..1 fleet-saturation %. But it scales the wrong direction for HPA — at high saturation HPA needs MORE pods, but desired = current × (metric/target) with metric=0.9, target=0.5 gives desired up. That's right. At idle metric=0 → desired=0 → min. Also right. Threshold becomes "trip-saturation level." (c) Stay with _can_accept_request but flip the math: count(livekit_egress_can_accept_request) - sum(livekit_egress_can_accept_request) This is "number of saturated pods." At idle = 0 (drains to min); at full saturation = current replicas (scales up to max). Replica-independent on the "scale up" leg. Is (a) the canonical pattern operators converge to? Or does (b)/(c) have hidden HPA-math gotchas I'm not seeing? And — is there a reason the LiveKit project hasn't exposed a fleet-level active_recordings_by_type Prometheus metric that would just work as a single-trigger HPA signal?
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!
3 Views
Open in Slack
PreviousNext
Powered by