https://livekit.io logo
Join Slack
Powered by
If i deploy my livekit agent to multiple VMS, how ...
# ask-ai
e
If i deploy my livekit agent to multiple VMS, how would i decide which vm gets picked. Ideally i want a round robin approach
✅ 1
t
Thanks for reaching out, Aditya. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about how LiveKit picks which VM/agent handles a job and how to get round‑robin behavior. Short answer • LiveKit already distributes jobs across agent workers using a round‑robin strategy by default when multiple workers share the same 
agent_name
. • That round‑robin is load‑aware: workers report load (default is CPU), and workers above the 
load_threshold
stop receiving new jobs. The dispatch also considers effective load and user locality (so selection is weighted toward under‑utilized and closer instances). What this means for your multi‑VM deployment • Deploy one or more agent worker processes on each VM and register them with the LiveKit server using the same 
agent_name
. The server will route new sessions to the registered workers in a round‑robin manner while avoiding overloaded workers. • If a VM becomes unhealthy or its worker load exceeds the threshold, it will be skipped automatically and jobs go to other workers. If you need stronger or custom round‑robin behavior • You can keep the default behavior if you want simple, balanced dispatch with load protection and locality. • If you need custom selection logic (for example, strictly pure round‑robin ignoring load, or selecting based on a custom metric), you can provide a custom 
load_fnc
when creating the worker so the server evaluates availability using your metric (for example GPU usage, free memory, or a custom score). Example concept: a. Register workers with the same 
agent_name
. b. Provide 
load_fnc
that returns a numeric load score. c. Tune 
load_threshold
so workers above that value stop receiving jobs. • In self‑hosted environments you’ll also want autoscaling (Kubernetes, Fly.io, etc.) and multiple workers per VM for high availability. Questions that will help me give exact steps • Are you on LiveKit Cloud or self‑hosted? (Cloud handles scaling for you.) • Do you want pure round‑robin regardless of load, or round‑robin that still respects worker load and locality? • Do you plan to use any custom metric (GPU, memory, custom service health) to decide eligibility? If you tell me which option you prefer and whether you’re self‑hosted or on LiveKit Cloud, I’ll give precise configuration examples and code snippets for registering workers and implementing a custom 
load_fnc
. Sources: An open source stack for real-time multimodal AI | LiveKit | Self-hosted deployments | LiveKit Documentation | Deploy and scale agents on LiveKit Cloud | LiveKit | Deploy and scale agents on LiveKit Cloud | LiveKit | Agents Framework | livekit/livekit_composite | DeepWiki, with interactive elements
e
i donot provide the agent name, its empty, will this work for that approach ?
t
Thanks for the follow-up — I’m replying about whether leaving 
agent_name
empty will still let you use round‑robin/load‑aware dispatch. Short answer • No — leaving 
agent_name
empty will not work for the named, round‑robin/load‑aware dispatch. You must set a non‑empty 
agent_name
for the server to route jobs among your agent workers. Why • The dispatch system uses 
agent_name
to identify the pool of workers that can handle a job. If 
agent_name
is omitted, the system falls back to automatic dispatch behavior and cannot perform explicit, load‑aware round‑robin distribution across a named worker pool. What you should do • Set the same non‑empty 
agent_name
for each agent worker instance you run across your VMs. That will enable the intended round‑robin with load awareness and locality handling. • Optionally provide a custom 
load_fnc
and tune 
load_threshold
if you want selection to consider GPU, memory, or any other metric instead of the default CPU‑based load. Quick questions to help me give exact steps • Are you on LiveKit Cloud or self‑hosted? (Cloud handles most of this automatically; self‑hosted needs you to register workers.) • Do you want strict round‑robin regardless of load, or round‑robin that still respects worker load/locality? If you tell me which platform you’re on and which behavior you prefer, I’ll give the exact config/code snippet to register workers with a shared 
agent_name
and an example 
load_fnc
if you want custom metrics. Sources: Agent dispatch | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation, with interactive elements
e
Both livekit and python agents are going to be self hosted
t
Great — thanks for confirming you’re self‑hosting both LiveKit and the Python agents. I’ll walk you through what to set and ask a couple quick questions so I can give exact config/snippets. Summary of what you need to do • You must set a non‑empty 
agent_name
on every agent worker instance. Use the same 
agent_name
for all workers you want grouped together. • LiveKit’s server will perform round‑robin dispatch across those workers while skipping workers that report load above 
load_threshold
. This gives you round‑robin behavior with built‑in load awareness and locality weighting. • For self‑hosted deployments, run multiple worker processes (one or more per VM), ensure they register with the LiveKit server, and optionally supply a custom 
load_fnc
if you want load decisions based on GPU, memory, or other metrics. Recommended steps (high level) 1. Choose an 
agent_name
for the pool (e.g., "voice-agent" or "transcribe-agent"). 2. Start agent worker(s) on each VM with that same 
agent_name
. 3. (Optional) Implement 
load_fnc
in your Python agent to return a numeric load score reflecting your preferred metric(s). 4. Tune 
load_threshold
(default 0.7) so high‑load workers stop receiving new jobs. 5. Run multiple workers across VMs and use autoscaling (K8s, Fly.io, etc.) for resilience. Example (what I can provide next) • A concrete Python snippet showing how to register a worker with 
agent_name
, add a 
load_fnc
, and set 
load_threshold
. • Kubernetes deployment hints (Deployment/HorizontalPodAutoscaler) and how many workers per VM to run for best availability. • A pure strict round‑robin option if you truly want to ignore load (I’ll show how to approximate that with a custom 
load_fnc
). Quick questions to tailor the instructions 1. Do you want strict round‑robin regardless of load, or round‑robin that still avoids overloaded machines? 2. Do you plan to use a specific metric for load (CPU is default), such as GPU utilization or free memory? 3. Would you like the Python code example or a Kubernetes manifest first? Tell me your preferences and I’ll give the exact Python config and deployment example you can copy/paste. , with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by