Hey, multiple workers are accepting the same livekit job, leading to race condition and multiple agents joined in the room for one participant

Thanks for reaching out, Siddharth. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

The issue is being caused in our prod environment, uat works completely fine, we have same agent name for all the workers that is true, we have now added num_idle_processes as 1 still the issue persists

Thanks — I appreciate the extra detail. I’ll help narrow this down so we can find what’s different in prod. Quick acknowledgement: you confirmed UAT is fine, all workers share the same

agent_name

, and you set

num_idle_processes

to 1 but the problem still occurs in production. That tells me the issue is environmental or deployment-related rather than a simple config flag. Please answer the following targeted questions so I can recommend precise fixes: 1. What LiveKit Agents version is running in production? 2. How are you deploying workers in prod (e.g., Kubernetes pods, VM instances, docker-compose)? How many pod/instance replicas exist for this agent service? 3. Does your orchestration / autoscaler ever start overlapping instances during deploys or restarts (brief windows with old + new pods both active)? 4. Do you run any sidecar or separate process that may also register with the same

agent_name

? 5. Share the sequence around accept/join in your agent code (a short snippet showing where

job_request.accept()

and the room join happen), and any retry/backoff logic around them. 6. How are access tokens created and scoped for job acceptance? Are tokens identical across workers or tied to a specific worker/process? 7. Any relevant prod logs around a duplicated acceptance (timestamps for both accepts, worker IDs, or connection IDs)? Based on your answers I can: • Point out specific code changes (idempotency guards, single-leader election, or token scoping). • Suggest deployment fixes (prevent overlapping instances during deploy, ensure single replica for agent_name, readiness/liveness probe tuning). • Provide log-driven diagnostics to identify whether both accepts truly originate from the LiveKit dispatch or from service-side logic. Which of the above details can you share first?

we have 2 pods for our workers

Thanks — that helps. Two pods for the same

agent_name

explains why multiple workers can accept the same job in prod. Here are concise actions you can take now (pick one path based on your HA needs): Options if you can run a single active worker • Reduce replicas to 1 for that agent service so only one process can accept jobs. • Ensure your deployment uses proper readiness/liveness probes and that rolling deploys avoid overlap (set

maxUnavailable: 0

,

maxSurge: 0

or use a preStop hook + graceful shutdown) so a job isn’t delivered to both old and new pods. Options if you need multiple pods for HA • Give each pod a unique

agent_name

so jobs are load-balanced across distinct agent groups instead of allowing duplicate acceptance. • Implement leader election within the pods so only the leader accepts jobs (e.g., use a simple lock in Redis/KV or Kubernetes leader election). • Add an idempotency/locking guard before calling

job_request.accept()

so only one instance can accept a job (check a short-lived distributed lock keyed by job/room ID). Other checks to add immediately • Confirm production tokens aren’t identical and scoped so multiple pods can independently accept the same job. • Inspect deploy/restart logs to find overlapping windows where both old and new pods are Ready. • Log the worker instance ID and timestamps when calling

accept()

to verify duplicate accepts and trace their origin. If you want, tell me which path you prefer (single active worker vs HA with multiple pods) and share: • Your deployment method (Kubernetes manifest or Helm values) and current replica settings, • The agent code snippet around

job_request.accept()

and join logic, so I can give exact manifest changes or a code example for a distributed lock or leader election.

But i can see logs in 1 pod only multiple agents joined the room, other pod does not have any logs, why is that

Thanks — I hear you: only one pod shows logs, yet that pod reports multiple agents joining the same room. Let’s narrow why a single pod is producing multiple agent joins. A few likely causes and quick checks (please tell me which apply or share the requested snippets): 1) Multiple agent processes inside the pod • Check whether the pod/container runs more than one agent process (e.g., multiple threads, worker child processes, or

num_idle_processes

> 1). Share your container command/args and the

num_idle_processes

value. 2) Single process creating multiple agent instances • Inspect your agent startup logic to see if it instantiates multiple agent handlers (for different queues or tasks). Share the short snippet where you initialize the agent and where

job_request.accept()

is called. 3) Duplicate accept/join calls in code paths • Retry, error, or callback logic might call

accept()

or re-join the room multiple times. Share the accept/join sequence and any retry/backoff logic. 4) Reconnects treated as new joins • If the agent thinks a prior join failed and retries, you may see multiple join logs from the same process. Share the join success/failure handling and timing of the log lines (timestamps). 5) Misleading logs (same process, different agent IDs) • Log what identifies each agent instance (process id, thread id, worker id, connection id). Add logging around

accept()

and room-join with a unique instance id to confirm whether these joins come from distinct agent instances or repeated attempts by one instance. What to share next (one or more): • Container command/args and env vars related to agents (including

num_idle_processes

). • The small code snippet showing

job_request.accept()

and the room join + retry handling. • The relevant log lines (timestamps, any IDs shown) from the pod that reports multiple joins. If you prefer, start by pasting the accept/join snippet and a representative log excerpt and I’ll point out exactly where to add idempotency or prevent duplicate joins.

I import os from livekit.agents import cli, WorkerOptions from lifecycle import entrypoint, prewarm from shared.utils.logger import get_logger logger = get_logger("orchestrator.main") # Prometheus metrics port (default: 9090, configurable via env) PROMETHEUS_PORT = int(os.getenv("PROMETHEUS_PORT", "9090")) if name == "__main__": logger.info("🚀 Starting Orchestrator...") logger.info("📊 Prometheus metrics will be exposed on port %s", PROMETHEUS_PORT) # Number of idle worker processes to keep warm # CRITICAL: Must be 1 to prevent duplicate job dispatch to same room # LiveKit dispatches room jobs to multiple workers if num_idle_processes > 1 # This causes multiple agents to join the same call NUM_IDLE_PROCESSES = int(os.getenv("NUM_IDLE_PROCESSES", "1")) cli.run_app( WorkerOptions( entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name="outbound-caller", port=8001, prometheus_port=PROMETHEUS_PORT, num_idle_processes=NUM_IDLE_PROCESSES, # Prevent duplicate job dispatch ) )

Thanks — this is helpful. You're running the orchestrator code with

num_idle_processes

set from env and two Kubernetes pods. You said only one pod's logs show multiple agent joins. To diagnose why one pod is creating multiple agent joins, please provide the following minimal items: 1. The prod pod's environment variables (or the

env:

section of the pod spec) showing

NUM_IDLE_PROCESSES

. 2. The container command/args / entrypoint as defined in the pod spec (so I can confirm the process is started with the same settings). 3. A short log excerpt from the pod that shows the multiple joins (include timestamps and any IDs printed). 4. Confirm whether the pod runs any sidecars or extra processes (print of

ps aux

from inside the pod is fine). 5. The code snippet where you call

job_request.accept()

and then join the room (if different from the earlier snippet). If you prefer, start by pasting (1) and (3) and I’ll look for whether the pod actually has multiple worker processes or if your single process is re-accepting/re-joining.

We do not have num_idle_processes in env, hence it should default to 1

Thanks — that helps. Since

NUM_IDLE_PROCESSES

isn’t set and should default to 1, let’s focus on runtime evidence inside the single pod that’s reporting multiple joins. Please provide the following minimal items so I can diagnose quickly: 1. A short log excerpt from that pod showing the multiple joins (include timestamps and any IDs shown). 2. The pod’s

env:

section and `command`/`args` from the Kubernetes manifest (or

kubectl describe pod

/

kubectl get pod -o yaml

output for that pod). 3. Output of

ps aux

from inside the pod (to confirm whether multiple agent processes are running). 4. If available, the code around

job_request.accept()

and the subsequent room-join (the accept/join snippet if it differs from what you shared). If you paste (1) and (3) first, I can often spot whether the single process is re-accepting/re-joining or whether multiple processes exist inside the pod.