https://livekit.io logo
Join Slack
Powered by
One task can handle 4 calls in our case, yet if we...
# ask-ai
r
One task can handle 4 calls in our case, yet if we have 2 tasks and try to place 8 concurrent calls at the same time it seems that only 4 go thru, the other 4 are being rejected, Which means all of them land on the same task. How do avoid that? how can we use livekit to auto balance?
✅ 1
t
Thanks for reaching out, luisa. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I understand you’re seeing only 4 calls succeed while the other 4 are rejected because they all land on the same task. I’ll explain what causes that and how to get LiveKit to auto-balance those calls across tasks. What’s happening • LiveKit assigns each job (room/call) to a single agent worker. If only one worker is available or the others are not registered/available, all jobs will land on that one worker and the rest will be rejected when it’s at capacity. How to fix it (what you can do) • Run multiple agent workers (tasks) that can accept jobs. LiveKit’s dispatcher will round-robin / distribute jobs across available workers automatically. • Ensure each worker is properly registered and healthy with the LiveKit server so they appear as available targets. • Use the agent load signalling: ◦ Workers signal load via 
load_fnc
(default: CPU) and stop accepting new jobs when load ≥ 
load_threshold
(default: 0.7). Confirm your workers’ load function and threshold are appropriate so busy workers are marked full and new jobs go to others. • For self-hosted: autoscale agent workers (e.g., Kubernetes HPA using CPU or custom metrics) so new workers are created before existing ones hit capacity. • For LiveKit Cloud: the platform will elastically scale agents for you — verify scaling settings and that new instances are coming up when load increases. • Verify networking/registration issues: if extra tasks are running but still not used, check logs and the LiveKit dashboard to confirm each worker registered successfully and isn’t being marked unhealthy or full immediately. Quick checklist to run now 1. Confirm you have multiple agent workers running and registered. 2. Check each worker’s health and load metrics (CPU, 
load_threshold
behavior). 3. Inspect logs for registration or network errors on the workers that are not receiving jobs. 4. If self-hosted, add autoscaling rules so new workers start before capacity is reached. If you want, tell me: • How you’re deploying agents (LiveKit Cloud or self-hosted/Kubernetes), • How many worker tasks you currently have running, • Any worker logs showing registration or “full” events. I’ll suggest the exact checks and config changes next based on that. Sources: Self-hosted deployments | LiveKit Documentation | Deploy and scale agents on LiveKit Cloud | LiveKit | Agents Framework | livekit/livekit_composite | DeepWiki | Deployment reliability on LiveKit Cloud | LiveKit, with interactive elements
r
self hosted but we are not using kubernetees, how can I confirm how many workers are registered?
t
Thanks — I’ll help you confirm how many agent workers are registered in your self‑hosted, non‑Kubernetes setup. Ways to check registered workers • LiveKit server /stats HTTP endpoint ◦ Query the server’s 
/stats
endpoint (usually 
http://<livekit-host>:7880/stats
) and look for agent/worker entries and their statuses. This shows active agents and basic metrics. • Server logs ◦ Check the LiveKit server logs for worker registration entries. Look for lines showing agents connecting/registered or any errors about agent registration or health checks. • Agent logs ◦ On each worker host, check the agent process logs for successful registration messages and for any “marked full” or unhealthy events. • Prometheus / metrics (if enabled) ◦ If you expose LiveKit metrics, inspect Prometheus for agent-related metrics (registered agents, load, or job assignments) to see how many are active. • LiveKit dashboard (if you run the UI) ◦ If you have the LiveKit dashboard running, it shows connected agents and their states. Quick checks to run now 1. Curl the server stats: 
curl http://<livekit-host>:7880/stats
and scan for agent entries. 2. Tail the LiveKit server log while restarting or starting a worker to watch registration messages. 3. On each worker host, 
journalctl -u <agent-service>
or check the agent log file for “registered” confirmations. If you’d like, tell me: • the LiveKit server host/port you can access (or paste a redacted 
/stats
output), and • a sample server or agent log snippet, and I’ll point to the exact lines to confirm registration and any issues. , with interactive elements
r
so I see 4 worker per agent in livekit, but I have two tasks running
t
Thanks — I’m responding to your note that you see 4 workers per agent in LiveKit but only two ECS tasks running (one Healthy, one Unknown), and calls are landing on the same task. What to check next (quick steps) 1. Confirm registered workers on the LiveKit server • Curl the server stats: 
curl http://<livekit-host>:7880/stats
and look for agent entries and worker counts. Share the redacted output if you want me to read it. 1. Verify each ECS task’s agent processes actually registered • On each task host, check agent logs for “registered” lines or connection errors. • For systemd/launcher logs: 
journalctl -u <agent-service>
or view the container logs in ECS for the agent process. • If one task’s health is “Unknown”, that task may not have completed registration or is failing health checks. 1. Check agent configuration values • Ensure all workers use the same 
agent_name
if you intend them to be pooled. • Confirm 
load_fnc
and 
load_threshold
settings (default: CPU & 0.7). If a worker is above threshold it will stop receiving new jobs. • Make sure each worker has unique process IDs but the same agent name (if you want them to be treated as separate workers under the same agent). 1. Inspect health/metrics and networking • Confirm ECS task networking allows outbound connection to the LiveKit server (no firewall/NACL blocking). • Look at LiveKit server logs to see if it marks any agent as unhealthy or rejects registrations. 1. Quick validation test • Restart one ECS task’s agent while tailing the LiveKit server logs to watch registration events — you should see the worker connect and the server assign jobs to it after it’s healthy. • After confirming both are registered as healthy, attempt the 8 concurrent calls again. If you want, paste: • The 
/stats
output (redact secrets) or • One agent log snippet from a task showing registration or errors I’ll point out the exact lines to confirm registration and why the “Unknown” health task isn’t receiving jobs. , with interactive elements
r
do the amount of workers available give us the load? we can take one load per worker?
t
Good question — I’m responding to your ask whether each worker equals one unit of load (one call). Short answer • No — worker count is not a 1:1 limit. LiveKit computes load per agent from resource usage (default: CPU) and uses that to accept or reject new jobs. A single worker can handle multiple concurrent calls depending on CPU/memory/network. What determines capacity • Default 
load_fnc
= CPU utilization (values 0–1) and 
load_threshold
default = 0.7. When a worker’s computed load ≥ threshold it stops accepting new jobs. • Recommended baseline: a 4‑core / 8GB agent can handle ~10–25 concurrent jobs depending on your workload (speech, recordings, codecs, etc.). If you want a strict per-worker job limit • Implement a custom load function that returns load = active_jobs / max_jobs (clamped to 1.0). That makes LiveKit treat each worker as able to handle up to 
max_jobs
. Sizing and autoscaling tips • Scale before hitting 
load_threshold
(e.g., scale at 0.5 if threshold is 0.7). • Monitor CPU, memory and active_jobs metrics and increase worker count or instance size if load approaches threshold. Next step • Would you like the exact example of a custom load function to enforce N calls per worker, or help interpreting your current 
/stats
output? Sources: Self-hosted deployments | LiveKit Documentation | Server options | LiveKit Documentation | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit, with interactive elements
r
Would you like the exact example of a custom load function to enforce N calls per worker, or help interpreting your current 
/stats
output? yes, so how do we know how many calls per worker do we assing? what does a worker do? do the stt and other things are shared? what are the best practices?
t
Thanks — I’ll answer each of your questions and give practical next steps. What a worker does • A worker (agent server process) registers with the LiveKit server, accepts job assignments, and runs each session as an isolated job process for the duration of that call. • Each job keeps its own audio buffers, STT/TTS/LLM context, and network connection — sessions are isolated to prevent interference. How many calls per worker • There’s no fixed 1:1 mapping. Capacity depends on CPU, memory, network, and which features you use (noise reduction, STT model size, TTS, recording, etc.). • As a starting guideline: a 4 vCPU / 8 GB instance often supports ~10–25 concurrent voice sessions, but this varies by workload. Are STT/LLM/TTS shared across workers or sessions? • Per-session resources are not shared across sessions on the same worker; each session has its own STT/LLM/TTS streams/contexts. • You can prewarm models on startup to reduce per-session overhead, but runtime contexts remain per-session. How load is measured and enforced • By default LiveKit uses a 
load_fnc
(CPU utilization) and 
load_threshold
(default 0.7). When computed load ≥ threshold, the worker stops accepting new jobs. • That means one worker can accept multiple calls until its measured load hits the threshold. If you want strict “N calls per worker” • Use a custom load function that reports load = active_jobs / N (capped at 1.0). Example:
Copy code
def compute_load(server):
    return min(len(server.active_jobs) / N, 1.0)
server.load_fnc = compute_load
Or TypeScript:
Copy code
const computeLoad = (server: AgentServer) => Math.min(server.activeJobs.length / N, 1);
server.loadFnc = computeLoad;
• With this, each worker will accept up to N concurrent jobs before being considered “full.” Best practices • Start with the 4 CPU / 8 GB baseline and run load tests reflecting your actual features. • Scale before threshold: autoscale at a lower point (e.g., 0.5) if threshold = 0.7 to avoid rejections. • Prewarm heavy models (VAD, STT, LLM) on worker startup to avoid spikes on first session. • Use observability: monitor CPU, memory, active_jobs, and 
/stats
to tune N and instance sizes. • Allow graceful drain on deploys (10+ minute grace for voice sessions). Next step I can help with • If you want, tell me the instance size and average CPU/memory during calls or paste a redacted 
/stats
output, and I’ll suggest an initial N and either a 
load_fnc
or sizing change to reach it. Sources: Server lifecycle | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Server options | LiveKit Documentation | Deploy and scale agents on LiveKit Cloud | LiveKit, with interactive elements
r
how about a 2core 8gb how many calls on average?
t
Short answer • A 2‑core / 8GB instance will support fewer concurrent calls than the recommended 4‑core baseline; there’s no exact guaranteed number. Expect roughly half the capacity of a 4‑core node — ballpark of a few (e.g., ~5–12) concurrent voice sessions depending on workload — but you must validate with tests. Why it varies • Capacity depends on CPU load from STT, noise reduction, VAD, TTS, recording, and any LLM work. Each session runs isolated processes and uses CPU/memory per session, so heavier features reduce concurrent capacity. Practical next steps (recommended) 1. Run a short load test with your real agent configuration (STT model, VAD, TTS, etc.) and measure CPU/memory per concurrent session. 2. Monitor 
/stats
(or metrics) and check worker CPU and active_jobs. Stop accepting new sessions when load approaches your 
load_threshold
. 3. If you want deterministic limits, use a custom 
load_fnc
that reports load = active_jobs / N to enforce N calls per worker. 4. Autoscale: scale up before the threshold (e.g., scale at 0.5 if threshold = 0.7). If you want, I can: • Provide the exact custom 
load_fnc
code (Python/TypeScript) to enforce N calls per worker, or • Help interpret your 
/stats
output or a short load-test result to pick an appropriate N. Which would you prefer? Sources: Self-hosted deployments | LiveKit Documentation | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit, with interactive elements
r
so by default if we don’t override the load_fnc and on request it’s gonna work with those defaults?
t
Yes — by default LiveKit will handle it using the built‑in behavior. What happens with defaults • LiveKit uses a CPU‑based 
load_fnc
and a 
load_threshold
of 0.7 by default. • Each worker reports a load value (0–1) based on CPU; when load ≥ 0.7 the worker stops accepting new jobs. • The server will distribute new sessions across registered, healthy workers based on those load signals (so jobs are auto-balanced without custom code). What you should still do • Verify your workers register and appear healthy (
/stats
or server logs). • Monitor CPU and active_jobs to ensure the default CPU metric reflects your actual capacity. • If you need a strict per‑worker job limit or your workload isn’t well represented by CPU alone, implement a custom 
load_fnc
(or autoscale) instead. If you want, I can provide the exact default behavior log lines to look for in 
/stats
or show a short custom 
load_fnc
example to enforce N calls per worker. Which would you like? , with interactive elements
r
so I should just remove that implementation if I want it to work with the livekit defaults, would that work still for self hosted?
t
Yes — removing your custom 
load_fnc
will revert the agent to LiveKit’s default CPU‑based load reporting (with a 
load_threshold
of 0.7). That default behavior works for self‑hosted deployments as long as your agents are correctly registered and healthy. Safe steps to switch back 1. Remove or disable your custom 
load_fnc
in the agent code/config. 2. Restart the agent processes so they register with the server using the default load function. 3. Verify registration and load reporting: ◦ Curl 
http://<livekit-host>:7880/stats
and confirm agents appear and report a 
load
value. ◦ Check agent and server logs for “registered” and health messages. 1. Monitor CPU and active_jobs for a few load cycles to ensure CPU reflects your true capacity and that sessions are being balanced. When to keep a custom function • If CPU doesn’t reflect your real constraint (e.g., you want to limit N concurrent calls per worker), implement a 
load_fnc
that uses active job count or a combined metric. If you want, I can: • Show the minimal change to remove your custom 
load_fnc
, or • Provide a short 
load_fnc
example that enforces N calls per worker. Which do you prefer?, with interactive elements
r
do I need to remove the on request function too?
t
You only need to remove/disable the custom 
load_fnc
. The 
on_request
(or request handler) can stay — it handles job setup logic, not load reporting. What to do 1. Remove/disable the custom 
load_fnc
so the agent falls back to LiveKit’s default CPU-based load reporting (threshold 0.7). 2. Keep your 
on_request
handler intact if it performs session setup (prewarm, model init, routing, etc.). 3. Restart the agent so it re-registers with the server using the default load function. 4. Verify with 
curl http://<livekit-host>:7880/stats
and agent/server logs that agents appear healthy and report 
load
values. If you want, paste your 
on_request
and 
load_fnc
snippets and I’ll show the minimal edit to remove only the 
load_fnc
. , with interactive elements
r
what’s the recommended setup to have if using python and subprocesses? It seems that it’s complicating things and I want to have a task with N workers, and no subprocess at the moment we haave 1 task 2 workers and 4 subprocesses
t
Thanks — I’ll keep this focused. Short answer • Don’t manually spawn subprocesses. Let the LiveKit agent framework manage job subprocesses. Run N worker processes (AgentServer instances) per task/host and let each worker spawn isolated job subprocesses automatically. Why • The framework already isolates each session in a child subprocess. Manual subprocess management adds complexity, causes registration/health confusion, and can interfere with load reporting and autoscaling. Recommended setup (Python) 1. Run one AgentServer per worker process (one process = one worker). 2. Keep your agent logic (the 
on_request
/ handler) in the AgentServer — do NOT call subprocess.Popen yourself. 3. Use a custom 
load_fnc
only if you need to limit by active job count; otherwise rely on the default CPU-based load function. 4. Prewarm heavy models in startup (setup/prewarm) so subprocesses don’t pay the cold-start cost. 5. Monitor 
/stats
, CPU, memory, and 
active_jobs
to size workers and autoscale. Minimal example (concept)
Copy code
server = AgentServer()
@server.rtc_session(agent_name="my-agent")
async def handler(ctx):
    # session logic (no manual subprocesses)
    ...
if __name__ == "__main__":
    server.start()
If you currently have 1 task → 2 workers → 4 manually spawned subprocesses, that likely means each worker is creating job subprocesses under load — which is expected. Remove any additional custom subprocess spawning so the SDK controls job lifecycle. Next step • If you want, paste your agent start code and any subprocess logic and I’ll show the minimal edits to remove manual subprocesses and run N workers cleanly. Sources: Agents Framework | livekit/livekit_composite | DeepWiki | Server lifecycle | LiveKit Documentation | Job lifecycle | LiveKit Documentation | Server options | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation, with interactive elements
r
how can I monitor stats, you mean via code?
t
Yes — you can monitor LiveKit stats both via HTTP endpoints and programmatically. Here’s how to do it and what to watch for. Quick checks (no code) • Server 
/stats
endpoint: curl 
http://<livekit-host>:7880/stats
to see registered agents, worker load values, active_jobs, and session counts. • Agent health endpoint: curl 
http://<agent-host>:8081/
— returns 200 when the agent is healthy. • Logs: check LiveKit server logs for agent registration lines and agent process/container logs for “registered”, “draining”, or error messages. Programmatic monitoring • Poll 
/stats
from your code (HTTP GET) and parse JSON to track: ◦ number of registered agents/workers ◦ 
load
value per worker ◦ 
active_jobs
(sessions) ◦ health status • Example (Python, minimal):
Copy code
import requests
resp = requests.get("http://<livekit-host>:7880/stats")
data = resp.json()
# inspect agents, active_jobs, and load values in data
Metrics & observability options • Expose and collect Prometheus metrics if you run a metrics stack; track CPU, memory, and active_jobs. • Use LiveKit Agent Observability (if available) or your dashboard to view session traces, transcripts, and per-job latency. • Monitor autoscaling triggers (CPU or custom metric) and 
load
relative to 
load_threshold
(default 0.7). What to watch for • Agents not listed in 
/stats
→ registration/network problem. • Workers showing 
load
≥ 0.7 → they will stop accepting new jobs. • High 
active_jobs
with high CPU → scale up or reduce per-worker concurrency. • Repeated “draining” or health failures in logs → investigate shutdown or startup issues. Next step • Tell me which you want: a Python script to poll 
/stats
regularly, a Prometheus scrape config example, or help interpreting a redacted 
/stats
output — I’ll provide the snippet. Sources: Server options | LiveKit Documentation | Understand and Improve Agent Latency | LiveKit | Agent deployment overview | LiveKit Documentation | LiveKit Agent Observability | Debug and optimize voice agents | LiveKit | Deployment reliability on LiveKit Cloud | LiveKit, with interactive elements
r
so we don’t need subprocesses at all?
t
Short answer: No — you should not spawn subprocesses yourself. Why • The LiveKit agent framework already runs each session as an isolated child subprocess and manages lifecycle, isolation, and registration. Manual subprocesses add complexity and can break load reporting, health checks, and autoscaling. What to do instead • Run one AgentServer process per worker (one worker = one process). • Put session logic in the agent handler (
on_request
/ 
rtc_session
) — do not call 
subprocess.Popen
. • Prewarm heavy models in startup/setup hooks so job subprocesses avoid cold starts. • Control concurrency with the default CPU-based 
load_fnc
or a custom 
load_fnc
(e.g., 
active_jobs / N
) and scale workers via your orchestration, not by spawning processes. Minimal Python pattern
Copy code
server = AgentServer()

@server.rtc_session(agent_name="my-agent")
async def handler(ctx):
    # session logic here — no subprocesses
    ...

server.start()
If you paste your current agent start code and subprocess snippet, I’ll show the exact edits to remove manual subprocesses and run N workers cleanly. , with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by