We have LiveKit RTC deployed on AKS across multipl...
# ask-ai
f
We have LiveKit RTC deployed on AKS across multiple regions. Multiple RTC nodes run behind a load balancer, with agent workers connecting to different nodes. Media and presence are propagated via a globally replicated Redis SFU. The challenge is reliably delivering new job requests of type JT to agents when the agent and client are connected to different RTC nodes. Example: Client connects to RTC Node 2. Agent is connected to RTC Node 1. A JT job is created for that agent, but intermittently the job request is not forwarded and the agent never receives it. Agents are registered by name, all join with job type JT, use TTL-based registration and heartbeats. We have ~40 agents at peak load. No obvious registration or idle-state issues, and no clear logs pointing to failures. Has anyone seen similar agent-job ownership or distributed signaling issues in multi-node, multi-region LiveKit setups? Any recommended patterns to ensure reliable cross-node job delivery?
t
Thanks for reaching out, Arnav. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your description of intermittent JT job requests not reaching agents when client and agent are on different RTC nodes. Quick diagnosis checklist and recommended patterns to ensure reliable cross-node job delivery: 1) Confirm a single shared Redis cluster for signaling • All RTC nodes and agent workers must use the same Redis instance/cluster for pub/sub and state. If nodes use region-local Redis replicas or partitioned Redis, job events can be missed. 2) Verify Redis pub/sub health and latency • Check Redis connection counts,
PUBSUB NUMSUB
for job channels, and any replication lag. High latency or partitioning causes missed or delayed dispatches. 3) Check agent registration visibility • Agents must publish global availability (name, JT type) to Redis. Ensure heartbeats interval < TTL and no network/firewall drops causing agents to be marked offline transiently. 4) Increase worker redundancy per agent • Run multiple agent worker instances for each agent name (recommended 2–3+) so if one worker is unavailable the job can be accepted by another. 5) Enable debug/trace logs on agent servers and RTC nodes • Capture: job published, job received by node, agent worker subscription, accept/reject, connect-to-room attempts. Look for silent handler errors or rejected jobs due to load checks. 6) Validate load/availability settings • Confirm
loadFunc
and
loadThreshold
aren’t causing agents to reject jobs unexpectedly under peak load. 7) Confirm dispatch flow end-to-end • Use webhooks/CLI or instrumented logs to assert these steps occur for a JT job: a. Job created on Node A b. Job event published to Redis c. Agent worker(s) on Node B receive event d. Agent accepts and connects to the room 8) Deployment patterns for multi-region • Prefer a single logical Redis (low-latency) rather than independent per-region Redis clusters. • Deploy agent workers in the same regions as RTC nodes to reduce heartbeat and connection issues. • Consider region-aware node selection for room allocation to reduce cross-region latency (doesn’t replace global Redis). Immediate diagnostic commands/checks you can run • Check Redis pub/sub subscribers for job channels (
PUBSUB NUMSUB <channel>
). • Inspect agent heartbeat timestamps in Redis or agent server metrics. • Temporarily enable debug logs on a pair of nodes (the node creating the job and the node hosting the agent) and reproduce a failing JT to capture the exact step where it’s dropped. If you’d like, tell me: • Redis topology (single global cluster vs per-region replicas) • Heartbeat interval and TTL values you’re using • Whether agents are single-worker-per-name or multiple workers I’ll suggest next focused checks or specific config changes based on those details. Sources: Distributed Setup | LiveKit docs | Server lifecycle | LiveKit docs | client-sdk-js | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki | Agent server options | LiveKit docs
f
000 I spoke to Akshit about this. 001 It's not a PUSub or Numsub issue. 004 That is only for application. 005 PUSub does not get replicated, and that's completely OK, primarily because we have two RTC instances, one in LATA, I mean two RTC clusters, let's put it that way, because it all, it all skills and applicants. 017 One in LTA, one in India, and there are LATA agents that register to the LATA. 023 RTEC and there are Indonesians that registered with the Indian RTC and it's completely fine. 028 That is intentional. 030 It is not supposed to register across two different nodes, so. 035 That is correct. 037 Like we won't change that. 039 And JT type to red is, yes, heartbeat intervals, all of that is good. 043 . 046 Akshad is right about the load function, though. 049 And the threshold that is coming, the 330 MB memory usage, we're gonna try to improve that and apparently it's been improved in the newer agents and newer life kit versions. 100 I'm gonna have to go through that and try to replicate it on that.
t
Thanks — I’m replying to your clarification that agent registration is intentionally per-region (two RTC clusters), Pub/Sub behavior is expected, and you suspect the load function / memory threshold (330 MB) may be contributing. A few targeted questions so I can recommend the most practical fixes: 1. Do the two clusters share the same Redis logical instance for agent/job state, or does each cluster use its own Redis? 2. Are JT jobs created by LiveKit’s built-in Agent framework (job API) on the client node, or via a separate external service that might forward jobs between clusters? 3. What exact LiveKit / agent versions run in each region? You mentioned newer agents have improved load behavior — are both clusters on the same version now? 4. How is your load calculated (briefly describe
loadFunc
) and how often is it sampled? 5. When a JT is missed, do debug logs show the job being published on the originating cluster? (If you haven’t enabled debug yet, enabling it on the nodes involved when reproducing will help.) Based on your answers I can recommend one or more concrete options (examples: a lightweight job-relay between clusters, raising/adjusting load thresholds, running extra workers per agent, or upgrading agent binaries) and exact config/log checks to run. Which of the five questions can you answer now?