https://livekit.io logo
Join Slack
Powered by
How to debug if large number of dispatch requests ...
# ask-ai
d
How to debug if large number of dispatch requests were created but agents only received 20% of them?
t
Thanks for reaching out, akshat. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I understand you’re asking how to debug why many dispatch requests were created but agents only received ~20% of them. Below are focused steps you can follow to find the root cause and resolve it. 1. Confirm agent capacity and workers • Ensure you’re running multiple workers per 
agent_name
. Single workers cause availability gaps and lost dispatches. • Check cluster/host counts and scale workers up if you see high dispatch volume. 1. Check agent registration & connectivity • Verify each agent server successfully registers with the LiveKit control plane and stays connected. • Look for network issues or firewall rules blocking agent ↔️ LiveKit communication. 1. Inspect agent load and resource usage • Monitor CPU, memory, disk, and network on agent hosts. High load or crashes can drop acceptances. • Don’t run other heavy apps on the same hosts as agents. 1. Review on_request / acceptance logic • If you implemented 
on_request
, confirm it calls 
req.accept()
(and not rejecting unintentionally). • If you rely on default behavior, know that agents auto-accept unless 
on_request
overrides it. 1. Look for timeouts during job startup • There’s a short window for agents to connect to a room after dispatch. Slow startup, heavy entrypoints, or network latency can cause timeouts and failed deliveries. 1. Use logs & observability • Enable DEBUG/INFO logs to track dispatch creation, routing, and acceptance events. • Check agent-side logs for rejected requests, errors, or crashes. • Use Agent Observability (if enabled) to inspect session traces and failed dispatches. 1. Verify dispatch configuration and rules • Confirm dispatch rules, agent names, and metadata match exactly. Misconfigured filters can route away many requests. • For room-based or SIP dispatches, ensure the correct 
RoomAgentDispatch
names and tokens are used. 1. Watch for queueing/backpressure and scale dynamically • Inspect dispatch queue depth and any backpressure metrics. • Implement auto-scaling (Kubernetes or similar) to add workers under high load. Diagnostic actions to run now 1. Pull agent logs for the time range when requests were created and search for connection failures, rejects, or crashes. 2. Count registered/available workers vs dispatched jobs at that time. 3. Check metrics: CPU/memory, network errors, and dispatch queue length. 4. Confirm whether 
on_request
is present and whether it accepted or rejected requests. If you can share: • whether you run multiple workers per agent name, • recent agent logs or error snippets, • and whether 
on_request
is implemented, I can help interpret logs and recommend targeted fixes. Sources: Agents Framework | livekit/livekit_composite | DeepWiki | How to Get Help from LiveKit | Agent server options | LiveKit docs | https://blog.livekit.io/streamline-troubleshooting-with-agent-observability/#/portal/signup| | Agent dispatch | LiveKit docs
d
I have request_fnc which publishes how many job requests were received whether accepted or rejected. But the total requests received by all running agents is 20% of all the dispatch requests created. In livekit server logs, I'm able to see logs like 
no workers with sufficient capacity
but only for 2% of the dispatch requests.
t
Thanks — I’m looking at two facts you gave: your 
request_fnc
shows agents received ~20% of dispatches, and server logs show 
no workers with sufficient capacity
for only ~2% of dispatches. To narrow this down I need a few details and will suggest targeted checks. Quick clarifying questions • How many workers are running per 
agent_name
during the incident? Any autoscaling in place? • Do you implement an 
on_request
handler in agents? If so, does it sometimes reject requests or short-circuit 
req.accept()
? • Are agents evenly distributed across hosts/regions, or concentrated on a few machines? • Do you see any agent-side errors or crashes in the agent logs for the same timeframe? • Are there any rate limits, proxies, or load balancers between LiveKit and agents? Immediate checks to run now 1. Correlate timestamps: Compare dispatch-created timestamps with agent log timestamps to see whether requests reached agents but were dropped/ignored. 2. Search agent logs for any rejected requests, exceptions, or long GC/startup pauses around dispatch times. 3. Count registered workers vs dispatched jobs at peak times (snapshot): if registered workers << concurrent dispatches, many requests may be queued/dropped before capacity checks log appears. 4. Inspect 
on_request
code paths for conditional rejections, early returns, or long synchronous work that prevents calling 
req.accept()
quickly. 5. Check network/connectivity metrics and LB/proxy logs for connection resets, timeouts, or 503s between control plane and agents. 6. Review resource metrics (CPU, memory, thread pools) for agents — look for saturation that would prevent request handling without emitting “no workers” on server. 7. Increase agent log level to DEBUG for a short window and capture a representative sample of dispatches. What to share if you want me to dig deeper • Small snippets from agent logs around a few missed dispatch timestamps. • The 
on_request
handler (or confirmation that default auto-accept is used). • Worker counts per 
agent_name
and any autoscaling config. • Any load balancer/proxy between LiveKit and agents. Send one or two log snippets and the 
on_request
info and I’ll correlate and point to the most likely root cause and next fix.
a
Bro are you using AgentServer @delightful-planet-44653?
t
Thanks — I want to clarify what you meant so I can help precisely. When you say "AgentServer @akshat", do you mean: • a specific AgentServer instance/process owned by a person or host named akshat, or • the AgentServer class/implementation you (or a teammate) wrote and annotated with @akshat, or • something else (eg. a Kubernetes pod/service labeled 
akshat
)? Also please confirm: • Are you running LiveKit’s AgentServer binary/library (official agent) or a custom agent implementation? • If it’s the official AgentServer, which version are you running? If you can answer those, please also attach: • A short agent log snippet around a missed dispatch timestamp, and • Your 
on_request
handler (or state that you use default auto-accept). With that I’ll correlate logs and code paths and suggest the next targeted checks.
d
No agent side errors or crashes. The first thing an agent does when an request is received is to publish that the request has been achieved, irrespective of whether it's going to reject it or not. Multiple instances of agents run ( > 50). Max dispatch requests created in a minute could be around 500-600 or 10 per second. Even if workers are not available the number of logs that say 
no workers
should match the number of dropped requests in livekit server logs. That count itself is 20% of the total dispatch requests created. Could be an issue with livekit server?
t
Thanks — I hear that: • Agents (>50) publish receipt immediately when they get a request and show no errors/crashes, • Agent-side received count is ~20% of dispatches (500–600/min peak), • LiveKit server logs show 
no workers with sufficient capacity
only for ~2% of dispatches, and • You suspect the LiveKit server may be dropping ~78% of dispatches before they appear to agents. This narrows possibilities. A few targeted checks and short diagnostics will quickly show whether the server is the culprit or something else is filtering/losing requests. Quick checks to run now 1. Correlate dispatch-created logs with server-side dispatch lifecycle • From the LiveKit server logs, pick a short timeframe (eg. 60–120s) with high dispatch volume. • For each dispatch created, confirm the server emitted any lifecycle log (queued/routed/assigned/failure). If many creations have no follow-up lifecycle logs, the server is dropping them before routing. 1. Check server-side rate limiting / queue/drop metrics • Look for any server metrics about dispatch queue depth, rejected/dropped dispatches, or internal rate limits. If these counters spike matching the missing percentage, the server is the source. 1. Search server logs for other failure reasons • Besides 
no workers with sufficient capacity
, search for errors like authentication failures, dispatch rule evaluation errors, datastore timeouts, or internal panics around the missing dispatch timestamps. 1. Verify dispatch routing & rule evaluation • Confirm the dispatch criteria/metadata used when creating dispatches exactly matches agent metadata and routing rules. If rules filter most requests, the server may drop or route them elsewhere without emitting the specific capacity message. 1. Confirm persistence/store acknowledgements • If LiveKit uses a backing store or queue for dispatches, check whether dispatch writes succeeded (ack/nack) in the server logs or store logs. 1. Increase server log level & capture a sample • Temporarily set LiveKit server logging to DEBUG for a 1–2 minute window at peak traffic and capture all dispatch-related logs. That will show why most dispatches never reach agents. Targeted data I can help analyze • A small sample (10–20) of dispatch IDs and their exact creation timestamps. • The LiveKit server log slice covering those timestamps. • Any server metrics about dispatch queue/rejections for that timeframe. • The dispatch creation payload (metadata/routing criteria) for a few examples. If you can share those (IDs + server log slice), I’ll correlate creation -> server lifecycle and point to the exact failure reason and next fix.
d
Running the default setup for self hosted livekit server with redis without any explicit or additional configurations. What could the issues?
t
Thanks — I’m responding to your note that you’re running a default self‑hosted LiveKit server with Redis and seeing ~20% of dispatches reach agents. Likely areas to check (focused, actionable) 1. Server-side dispatch queue / rate limits • Verify LiveKit’s dispatch queue depth and any counters for dropped/rejected dispatches. Default builds may have internal backpressure that drops requests under burst load. 1. Redis performance and availability • Check Redis CPU, memory, latency (PING, command latency) and whether it’s hitting maxclients or eviction. Slow or failing Redis ops can cause the server to fail to persist/route many dispatches. 1. Server logs at DEBUG during a peak • Temporarily enable DEBUG for 1–2 minutes at peak traffic and capture all dispatch lifecycle logs. Correlate dispatch-created timestamps with any follow-up lifecycle messages. If creations have no follow-up, the server is likely not enqueuing/routing them. 1. Dispatch rule evaluation / metadata mismatch • Confirm the dispatch payload metadata exactly matches agent metadata and any routing rules. If most dispatches are filtered, they won’t reach agents even though they were created. 1. Resource saturation on LiveKit server • Check CPU, memory, file descriptors, and network for the LiveKit server process. If the process is starved, it may silently drop work before emitting “no workers” logs. 1. Redis-backed enqueue/write errors • Search server logs for Redis write errors, timeouts, or rejections during the spike window. If writes to the dispatch store failed, dispatches may be lost. 1. Transport/proxy/load balancer between server and agents • If any LB or proxy exists, check its connection limits and timeouts. Even default setups may have system limits (ulimits, ephemeral ports) that cause silent drops. Immediate diagnostic steps you can run now 1. Capture a 60–120s DEBUG log window on the LiveKit server during a burst. 2. Capture Redis INFO and slowlog output for the same window. 3. Provide 10–20 dispatch IDs + exact creation timestamps and corresponding server log slices. 4. Snapshot LiveKit server resource metrics (CPU, mem, open FDs) at peak. If you share the logs/IDs and Redis INFO output, I will correlate creation → server lifecycle and point to the precise failure (eg. Redis timeouts, server queue drops, rule filtering).
a
@delightful-planet-44653 its with AgentServer, try autoscaling you Livekit server. It will fix the issue
t
Thanks — I hear your suggestion to autoscale the LiveKit server and that you’re running AgentServer. Autoscaling the LiveKit server is a reasonable next step; here’s a concise plan to try it and verify it fixes the missing dispatches. Why autoscaling can help • If the LiveKit server is CPU/memory/IO constrained or its dispatch queue/backpressure is being hit under bursts, adding server capacity reduces dropped or unrouted dispatches. Action plan to autoscale and validate 1. Autoscale setup • Deploy multiple LiveKit server instances (horizontal scale) behind your load balancer. • Use Kubernetes (or your orchestration) to autoscale based on CPU, memory, or a custom metric (dispatch queue depth or Redis latency). • Ensure consistent config across replicas (same Redis, DB, and dispatch routing config). 1. Adjust system limits and networking • Raise ulimits/open file descriptors and ephemeral port ranges on hosts to avoid connection exhaustion under burst load. • Ensure the LB/proxy can handle increased connections and isn’t imposing low connection/timeouts. 1. Tune Redis and persistence • Ensure Redis can handle higher throughput (sufficient CPU/memory, configure maxclients, monitor command latency). Autoscaling LiveKit without ensuring Redis capacity will shift the bottleneck to Redis. 1. Short validation test • Reproduce a burst (500–600/min or ~10/s) while scaling up LiveKit replicas. • Capture server DEBUG logs and agent receipts for that window. 1. Verification checks (did autoscale fix it?) • Agent receipt count should rise near 100% of dispatches. • LiveKit server logs should no longer show many dispatches with no lifecycle follow-up. • The 
no workers with sufficient capacity
log should be rare and match actual capacity issues only. • Redis metrics should remain stable (no slow commands or errors). If autoscaling does not close the gap • Collect a sample of dispatch IDs and server log slices for those timestamps, plus Redis INFO/slowlog and LiveKit server resource metrics. I’ll correlate them and point to the remaining bottleneck. If you want, tell me: • how you currently deploy LiveKit (K8s, VM, ECS), and • the autoscaling metric you plan to use (CPU, mem, or a custom metric) — and I’ll give a short checklist tuned to that environment.
Open in Slack
PreviousNext
Powered by