how many calls can livekit worker handle simultaneously

Thanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

when dispatching using create_dispatch method, can we put the request in a queue and let the worker poll the queue and get the dispatch request, is it possible?

Thanks — you're asking whether you can place requests in a queue when calling

create_dispatch

and have workers poll that queue to pick up dispatches. Short answer: No—you cannot directly place dispatch jobs into a user-visible queue for workers to poll.

create_dispatch

is an immediate operation that submits a job to LiveKit, and the LiveKit server handles job queuing and delivery. What actually happens: • When you call

create_dispatch

, the server creates the job and places it into LiveKit’s internal dispatch queue. • Workers register with the server and receive jobs automatically over their persistent connection. • A worker accepts a delivered job by calling something like

job_request.accept()

inside its job handler. You do not need to implement manual polling in the worker. If you need different behavior: • If you want explicit, user-controlled queuing (custom priorities, delayed delivery, custom retry logic, etc.), implement an external queue (Redis, RabbitMQ, SQS, etc.) in front of

create_dispatch

. Your dispatcher would pop a job from that external queue and call

create_dispatch

when you want the LiveKit server to handle assignment. • For typical use, rely on LiveKit’s internal dispatch queue—workers already act like pollers by staying registered and receiving jobs when available. If helpful, tell me: • whether you need custom priorities, delayed retries, or visibility into queued jobs, and I’ll suggest a concrete pattern for integrating an external queue with

create_dispatch

. Sources: An open source stack for real-time multimodal AI | Server lifecycle | LiveKit Documentation | Agent dispatch | LiveKit Documentation

i need all the above that is: custom priorities, delayed retries, or visibility into queued jobs,

Thanks — you're asking for a design that supports custom priorities, delayed retries, and visibility into queued jobs when dispatching to LiveKit agents. That’s clear; here’s a practical, production-ready pattern you can use and the concrete steps to implement it. High-level approach • LiveKit’s built-in dispatch is low-latency and reliable but does not support priority queuing, delayed retries, or queue visibility. • To get those features, put an external, monitorable queue and a dispatcher service in front of LiveKit’s

create_dispatch

/ AgentDispatchService API. The external system manages priorities, retry scheduling, and observability; the dispatcher calls LiveKit only when a job is ready to be assigned. Recommended architecture Room created / incoming call → enqueue dispatch request (with metadata) → priority queue + retry scheduler → dispatcher service (consumes queue) → call LiveKit AgentDispatchService / create_dispatch → LiveKit server assigns job to agents → worker accepts job What each component does • Client / Ingest: When a room or call is created, publish a dispatch message to your queue rather than calling LiveKit directly. Include metadata: priority, customer tier, timeout, attempt count, correlation IDs. • Priority Queue: Use a queue that supports priorities and visibility: Redis (ZSET or Streams), RabbitMQ (multiple priority queues), Kafka (separate topics per priority), or SQS with priority modeling. This gives you control over order and visibility into pending jobs. • Retry Scheduler / Backoff: Use a scheduler (Celery, Temporal, Sidekiq, Cloud Tasks, or a custom worker) to re-enqueue failed jobs with exponential or fixed backoff. Store attempt count & last error in the job payload. • Dispatcher Service: A small service that: a. Consumes the highest-priority ready job. b. Calls LiveKit’s

create_dispatch

(AgentDispatchService API) with the job metadata. c. If LiveKit returns immediate failure (no available agents, rejects), the dispatcher decides whether to re-enqueue with delay or mark as failed based on attempts and backoff policy. d. Emits logs/metrics for lifecycle events: queued → dispatched → started → completed → failed. • Observability & Dashboard: Export queue depth, job age, retry counts, and LiveKit dispatch statuses to your monitoring stack (Prometheus/Grafana, Datadog). Provide a dashboard and search by correlation ID or room ID. • Worker behavior: Keep workers registered with LiveKit normally. Worker job handler calls

job_request.accept()

when ready. Ensure workers gracefully handle shutdown with a grace period to let in-progress calls finish. Concrete implementation steps 1. Choose queue and scheduler ◦ For priorities + visibility: Redis Streams or RabbitMQ are simple; Kafka or Google Cloud Tasks for large scale. ◦ For delayed retries: choose a scheduler with native delay (Cloud Tasks, Celery ETA), or use delayed queues (Redis sorted sets with score=ready_at). 1. Define job schema (example fields) ◦ job_id, room_id, priority, attempts, max_attempts, earliest_dispatch_time, correlation_id, metadata (customer tier, codec, features) 1. Enqueue on room creation ◦ Push job to queue with priority tag and earliest_dispatch_time = now. 1. Dispatcher logic (pseudocode) ◦ Poll highest-priority job where earliest_dispatch_time <= now. ◦ Call LiveKit

create_dispatch

with job metadata. ◦ If dispatch accepted by LiveKit: mark job started; update metrics. ◦ If dispatch rejected or no agents available: ▪︎ If attempts < max_attempts: increment attempts, set earliest_dispatch_time = now + backoff(attempts), re-enqueue. ▪︎ Else: mark job failed and notify upstream systems. 1. Logging & metrics ◦ Emit events for: enqueued, dispatched, accept, reject, retry, failed, completed. ◦ Track queue depth, average wait time, per-priority backlog. 1. Edge cases & recommendations ◦ Correlate job_id with LiveKit dispatch IDs to trace end‑to‑end. ◦ Cap queue retention for stale jobs and implement alerting for high backlog. ◦ Avoid immediate re-dispatch loops — use a backoff to prevent flooding LiveKit. ◦ For SLA-sensitive work, reserve a worker pool (or an agent_name) for high-priority jobs. Why this fits LiveKit • LiveKit already provides an internal dispatch queue and reliable job delivery, but it does not expose priority/delayed-retry controls or queue telemetry. • The external queue plus dispatcher pattern preserves LiveKit’s job assignment mechanism while adding the control and visibility you need. If you want, I can: • Recommend specific technology choices based on your scale (expected concurrency, latency targets). • Provide an example dispatcher implementation (sample code) for Redis Streams + Python Celery or RabbitMQ + Node.js. • Draft a job schema and metrics dashboard panel list to start monitoring. Which of those would be most helpful next — tech stack recommendation, sample code, or a metrics/dashboard template? Sources: How Playback built a global stadium for watching sports together | LiveKit | Self-hosted deployments | LiveKit Documentation | Why do jobs stop after agent rejection? | LiveKit | Streamline troubleshooting with Agent Observability | Streamline troubleshooting with Agent Observability

understood, i have read about livekits load aware dispatching , there we cand efien a load fucntion based on which livekit can handle dispatches based on load , how can we leverage that? in that case wont we need to make dispatch call?

Thanks — I’ll explain how to leverage LiveKit’s load-aware dispatching, how it fits into your dispatch flow, and how to combine it with an external queue for priorities/delayed retries/visibility. Short answer • Load-aware dispatching lets an agent server advertise its current capacity (a float 0.0–1.0) using a

load_fnc

and

load_threshold

. LiveKit will only assign jobs to servers whose reported load is below the threshold. • You still need to issue dispatches (automatic or explicit). Load-aware dispatching is a gate that accepts or rejects assignments based on server load — it does not replace the dispatch call or an external queuing layer. How it works in practice 1. Agent server defines a load function ◦ Implement a function that returns a float 0..1 representing current load (for example: active_jobs / max_concurrency). ◦ Configure

server.load_fnc = compute_load

and

server.load_threshold = 0.9

. ◦ When LiveKit attempts to assign a job, it checks this value; if load >= threshold, the server will not receive the job. 1. Automatic vs explicit dispatch ◦ Automatic dispatch: If you rely on LiveKit’s automatic assignment (no

agent_name

), LiveKit will place jobs on servers that are reported as below their thresholds. ◦ Explicit dispatch: If you call

create_dispatch

(or use AgentDispatchService) targeting an

agent_name

, LiveKit will route the job to that agent group — but the server still must be under its load threshold to accept it. ◦ In both cases, the dispatch call is required to create the job; load-aware behavior only affects whether the job is assigned to a particular server. When to use load-aware dispatching vs an external queue • Use load-aware dispatching when: ◦ You want servers to self-throttle acceptance of new jobs based on real-time metrics (CPU, job count, custom signals). ◦ You want LiveKit to perform low-latency assignment while avoiding overloading busy agents. • Use an external queue when you need: ◦ Custom priorities, delayed retries, detailed visibility/monitoring, or complex retry/backoff policies. ◦ Fine-grained control over when to call

create_dispatch

(e.g., only call when a job is due for dispatch). Recommended combined pattern (best of both worlds) 1. Enqueue all incoming dispatch requests in your external queue with priority/visibility metadata. 2. Dispatcher picks the top job and attempts to dispatch to LiveKit via

create_dispatch

(explicit) or allows automatic dispatch (no agent_name). 3. LiveKit will route the job to servers that report load <

load_threshold

(because agents run your

load_fnc

). 4. If LiveKit cannot assign (no servers available or servers above threshold), the dispatcher: ◦ Re-enqueues the job with backoff (delayed retry), or ◦ Tries a different agent pool (if you have reserved pools for high-priority jobs). 1. When an agent accepts the job, it calls

job_request.accept()

and proceeds. Practical notes and tips • Design your

load_fnc

to represent the dimension that matters to you (concurrency, CPU%, memory pressure, GPU usage, or a combination normalized to 0..1). • Keep

load_fnc

cheap to compute and update frequently enough to reflect real capacity. • Set

load_threshold

with some headroom to avoid saturating the server (e.g., 0.8–0.9 depending on your safety margin). • Correlate dispatcher job IDs with LiveKit dispatch IDs for observability and tracing. • In LiveKit Cloud, you cannot customize

load_fnc

— this is only available in self-hosted deployments. If you want, I can: • Suggest a concrete

load_fnc

based on the metrics you care about (CPU, active_jobs, etc.). • Provide example dispatcher pseudocode showing how to handle LiveKit assignment failures and re-enqueue with backoff. • Recommend queue technologies and how to structure job metadata for easy tracing. Which would help you next — a sample

load_fnc

for your environment, or dispatcher pseudocode that integrates queue + LiveKit dispatch? Sources: Server options | LiveKit Documentation | Agent dispatch | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation

Currently we are using livekit agent framework built on python, which runs 4 workers, and its deployed on aws eks, with 2 active pods for now which can increase in future, so the load function works per pod basis, per worker basis or something else?

Thanks — great details. You’re asking whether the

load_fnc

is evaluated per pod, per worker, or something else for your Python agent framework running 4 workers inside pods on EKS. Here’s how it behaves and how you should think about it for scaling. Short answer • The load function is evaluated per AgentServer process instance — i.e., per pod/process that runs the AgentServer. It reflects that server’s view of its own capacity (not a global cluster view). • If a pod runs multiple worker "agents" inside the same AgentServer process, your

load_fnc

should aggregate across those workers (for example, use active job count across all workers in that process). If you run one AgentServer per worker in separate processes, each process reports its own load independently. How this maps to your setup • You said your Python agent framework runs 4 workers. If those 4 workers are created and managed by a single AgentServer instance in the pod (the common pattern), then: ◦ The pod’s AgentServer runs a single

load_fnc

. ◦ That function should compute load across all 4 workers (active jobs, CPU, memory, etc.) and return a single 0.0–1.0 value representing the pod/process capacity. ◦ LiveKit will consider that pod/process’s reported load when assigning jobs. • If your architecture instead runs separate AgentServer processes for each worker (one process per worker), each process reports its own load independently — LiveKit sees multiple servers and assigns jobs accordingly. Practical guidance • Confirm process layout: determine whether your 4 workers are threads/tasks inside one AgentServer process or four distinct AgentServer processes. This determines whether

load_fnc

is per-pod (single) or per-worker (multiple). • Implement

load_fnc

to reflect the true contention surface for that process. Example signals to include: ◦ number of active jobs in that AgentServer instance (divide by target max concurrency) ◦ CPU utilization and memory pressure of the process or pod (normalized to 0..1) ◦ custom indicators such as whether the process is running special CPU‑heavy features (noise cancellation, detectors) • Set

load_threshold

with headroom (e.g., 0.8–0.9) so the pod doesn’t saturate. • When scaling horizontally (adding more pods), LiveKit will see each pod/process as a separate server reporting its own load and will prefer servers under threshold. If you want, I can: • Suggest a concrete

load_fnc

example that aggregates active jobs and CPU% for a single AgentServer process. • Help you verify whether your current deployment runs one AgentServer per pod or one per worker (tell me how you spawn the 4 workers in your Python app). Which would you like?

Its process, right now on one pod, 4 processes run for worker

Thanks — this is helpful detail. Because you run 4 separate AgentServer processes (workers) in the same pod, here’s exactly how load-aware dispatching behaves and what to do. How load is evaluated • Load is evaluated per AgentServer process instance. Each of your 4 processes reports its own

load_fnc

value independently. • LiveKit treats each process as a separate server when assigning jobs. It will only route a job to a process whose reported load is below its configured

load_threshold

. What that means for your pod with 4 processes • Even though the processes share the same Kubernetes pod and host resources, LiveKit does not automatically aggregate their load — each process must report its own view. • If a single process reports high load, LiveKit will avoid dispatching to that process but may dispatch to other lower‑load processes in the same pod (if they report lower load). Recommended

load_fnc

design (per process) • Make

load_fnc

reflect the true contention surface for that process. Good signals: ◦ active job count for that process (e.g., len(active_jobs) / max_concurrency) ◦ process-level CPU% and memory usage (normalized to 0..1) ◦ process flags for heavy features (noise cancel, detectors) that increase cost • Combine signals into a single 0.0–1.0 value. Keep computation cheap and deterministic. Simple example (conceptual Python)

def compute_load(server):
    # weight job count higher, CPU as supporting signal
    job_load = min(len(server.active_jobs) / MAX_JOBS_PER_PROCESS, 1.0)
    cpu_load = min(get_process_cpu_percent() / 100.0, 1.0)
    # weighted sum, tuned to your environment
    return min(0.8 * job_load + 0.2 * cpu_load, 1.0)

Practical tips • Tune MAX_JOBS_PER_PROCESS to match what that process can handle (use load tests). • Set

load_threshold

with headroom (e.g., 0.8–0.9) to avoid saturation. • Monitor process-level metrics (CPU/memory, active_jobs) per process — not just per pod — so you can see which AgentServer instances are overloaded. • Correlate LiveKit dispatch IDs with process telemetry to trace assignment decisions. • If processes contend heavily for the same CPU/memory, consider one of: ◦ switching to a single AgentServer process that manages multiple workers and using one aggregated

load_fnc

, or ◦ running fewer processes per pod (or separate pods) so pod-level resource limits align with process-level reporting. Scaling implications • When you scale pods horizontally, LiveKit will see more AgentServer instances (processes) and distribute work across them according to each reported load. • Ensure your Kubernetes resource limits/requests are sized so processes can accurately measure and report CPU/memory. If you’d like, I can: • Suggest specific weights or thresholds based on your expected concurrency or CPU targets. • Provide a short script to collect process CPU% and active job count for use in

load_fnc

. Which would be most helpful next?

Do livekit server determine the agent realtime, or does it continuosly poll and pick the worker precomputed?

Good question — here’s how LiveKit’s dispatching works in practice. • LiveKit performs assignment in real time when a dispatch request is created. When you call

create_dispatch

(or LiveKit’s automatic dispatch path), the server immediately tries to route that job to an available agent server instance. • Agent servers do not need to "poll" an exposed queue. Instead, agents maintain persistent connections to the LiveKit server and the server pushes job requests to them. From an agent’s perspective this looks like it’s being delivered over an open channel rather than the agent continuously polling for jobs. • Load-aware behavior is evaluated at assignment time. Before assigning a job to a particular agent process, LiveKit checks that agent’s current reported load (the

load_fnc

value) and will only assign if that value is below the agent’s

load_threshold

. So the server makes a realtime decision using the most recent reported load from each agent. • If no suitable agent is available (all reported loads are at/above threshold or agents are disconnected), the dispatch attempt will fail or be rejected. That is why many teams place an external queue in front of LiveKit: the dispatcher repeatedly (or on schedule) tries again later to call

create_dispatch

when the queue logic decides an attempt should be made (delayed retry, priority order, etc.). In short: • Assignment is done in real time by the LiveKit server at dispatch time. • Agents do not poll a public queue; they keep persistent connections and receive jobs pushed from LiveKit. • Load checks are evaluated at the moment of assignment, so combining an external queue with load-aware agents gives you controlled retry/delay and LiveKit’s realtime routing when capacity exists. If you want, I can provide a short sequence diagram or dispatcher pseudocode showing how an external queue and realtime LiveKit assignment interact. Which would help you next?