we have more than enough self-hosted LiveKit pods, yet some of them got way more sessions than others, caused some to crash while others were almost empty. why this happened? how do you distribute those sessions?

Thanks for reaching out, Mohammed. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

autoscaler works fine, and we have way more pods than what we really needed, yet some got targeted with sessions till they crash and others were almost idle

Thanks — I hear you: autoscaling and overall pod count look fine, yet some pods keep getting overloaded and crash while others stay nearly idle. I’ll explain the most likely causes and give clear, actionable steps you can take to diagnose and fix this. Why this happens • LiveKit uses a stateful session assignment model that favors session continuity over purely round‑robin distribution. Once an agent is handling sessions, subsequent assignments can still land on it because of session affinity and the agent’s reported “effective load.” • The dispatch decision uses each AgentServer’s

load_fnc

(a 0–1 load score) and

load_threshold

(default 0.7). If the server’s reported load is below threshold it will accept jobs; if it’s at/above threshold it stops accepting new jobs. • If your autoscaler uses different metrics or reacts too late relative to the agent’s `load_fnc`/`load_threshold`, new sessions may keep landing on already‑busy pods until they crash. • Long‑running or resource‑heavy sessions (voice/video with extra processing) can tie up capacity and cause some agents to accumulate many long‑lived jobs. • Draining/termination behavior or slow reassignment after a failed accept can also let traffic pile onto a few pods instead of spreading out. Immediate checks to run • Confirm the

load_threshold

value you’re using (default is 0.7). • Check whether you use a custom

load_fnc

or the default (default uses CPU). • Inspect pod logs for job‑accept timeouts, rejected assignments, or repeated reassign attempts. • Look at per-pod metrics: CPU, memory, active session count, and assignment/reject events over time to see the pattern when spikes occur. • Verify termination/drain settings (grace period and draining behavior) on pods that crashed or were removed. Concrete actions to reduce imbalance 1. Align autoscaler metric with the agent load function ◦ Ensure the autoscaler scales on the same metric(s)

load_fnc

uses (CPU by default). If you use CPU for

load_fnc

, the autoscaler should also use CPU. 1. Scale earlier than the agent threshold ◦ If

load_threshold

= 0.7, set autoscaler scale-up trigger lower (e.g., ~0.5–0.6) so new pods come up before existing ones saturate. 1. Use a job-aware

load_fnc

(if you control it) ◦ Replace CPU-only load with a function that factors in active session count or weighted job cost so the dispatcher sees true capacity and spreads new jobs earlier. 1. Reduce chance of hot-piling on single pods ◦ Lower

load_threshold

so servers mark themselves “full” earlier, or limit concurrent jobs per agent process. 1. Tune scaling cooldowns and draining ◦ Reduce scale-up stabilization to react faster on spikes. ◦ Increase scale-down cooldown and ensure graceful draining so sessions finish rather than being abruptly reassigned. 1. Separate workloads ◦ If you have mixed short and long-lived sessions, consider separate agent pools/node groups tuned for each workload type. 1. Watch for stuck/long-running jobs ◦ Identify and fix jobs that leak memory or never finish; these can silently reduce capacity and concentrate new assignments on fewer healthy pods. 1. Ensure multiple workers per agent_name ◦ Run at least 3+ workers for each

agent_name

to allow natural failover and better distribution. If you’d like, I can help next with any of the following: • Review your current values: what

load_threshold

you’re using, whether you run a custom

load_fnc

, and what metric your autoscaler uses. • Suggest a sample

load_fnc

that includes active session count and CPU. • Provide a checklist of logs/metrics to gather (pod CPU, memory, active_sessions, assignment_rejects, job_accept_timeouts) so we can pinpoint where assignments are skewing. Which would you prefer me to help with first, and can you share: • your

load_threshold

(if customized), • whether you use a custom

load_fnc

, and • the type of sessions most affected (long voice sessions, short calls, video, etc.)? Sources: Deploy and scale agents on LiveKit Cloud | Deploy and scale agents on LiveKit Cloud | Self-hosted deployments | LiveKit Documentation | Server options | LiveKit Documentation | Agents Framework | livekit/livekit_composite | DeepWiki