freezing-forest-56439
12/19/2025, 6:23 AMtall-airline-16756
12/19/2025, 6:23 AMtall-airline-16756
12/19/2025, 6:23 AMPUBSUB NUMSUB for job channels, and any replication lag. High latency or partitioning causes missed or delayed dispatches.
3) Check agent registration visibility
• Agents must publish global availability (name, JT type) to Redis. Ensure heartbeats interval < TTL and no network/firewall drops causing agents to be marked offline transiently.
4) Increase worker redundancy per agent
• Run multiple agent worker instances for each agent name (recommended 2–3+) so if one worker is unavailable the job can be accepted by another.
5) Enable debug/trace logs on agent servers and RTC nodes
• Capture: job published, job received by node, agent worker subscription, accept/reject, connect-to-room attempts. Look for silent handler errors or rejected jobs due to load checks.
6) Validate load/availability settings
• Confirm loadFunc and loadThreshold aren’t causing agents to reject jobs unexpectedly under peak load.
7) Confirm dispatch flow end-to-end
• Use webhooks/CLI or instrumented logs to assert these steps occur for a JT job:
a. Job created on Node A
b. Job event published to Redis
c. Agent worker(s) on Node B receive event
d. Agent accepts and connects to the room
8) Deployment patterns for multi-region
• Prefer a single logical Redis (low-latency) rather than independent per-region Redis clusters.
• Deploy agent workers in the same regions as RTC nodes to reduce heartbeat and connection issues.
• Consider region-aware node selection for room allocation to reduce cross-region latency (doesn’t replace global Redis).
Immediate diagnostic commands/checks you can run
• Check Redis pub/sub subscribers for job channels (PUBSUB NUMSUB <channel>).
• Inspect agent heartbeat timestamps in Redis or agent server metrics.
• Temporarily enable debug logs on a pair of nodes (the node creating the job and the node hosting the agent) and reproduce a failing JT to capture the exact step where it’s dropped.
If you’d like, tell me:
• Redis topology (single global cluster vs per-region replicas)
• Heartbeat interval and TTL values you’re using
• Whether agents are single-worker-per-name or multiple workers
I’ll suggest next focused checks or specific config changes based on those details.
Sources: Distributed Setup | LiveKit docs | Server lifecycle | LiveKit docs | client-sdk-js | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki | Agent server options | LiveKit docsfreezing-forest-56439
12/19/2025, 7:53 AMtall-airline-16756
12/19/2025, 7:53 AMloadFunc) and how often is it sampled?
5. When a JT is missed, do debug logs show the job being published on the originating cluster? (If you haven’t enabled debug yet, enabling it on the nodes involved when reproducing will help.)
Based on your answers I can recommend one or more concrete options (examples: a lightweight job-relay between clusters, raising/adjusting load thresholds, running extra workers per agent, or upgrading agent binaries) and exact config/log checks to run. Which of the five questions can you answer now?