https://livekit.io logo
Join Slack
Powered by
Our participant agents aren't launching when a new...
# ask-ai
e
Our participant agents aren't launching when a new participant joins the room.
t
Thanks for reaching out, Josh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m sorry you’re seeing participant agents not start when someone joins the room. I’ll help you diagnose this step-by-step so we can pinpoint the cause. First, a few quick clarifying questions — the answers will narrow the likely causes: 1. Are you using automatic dispatch or explicit dispatch for agents? ◦ If you set an 
agent_name
in the agent entrypoint, automatic dispatch is disabled and agents must be explicitly dispatched. 1. How are agents configured to be dispatched for the room? (choose one) ◦ Via a participation token using 
RoomConfiguration.agents
◦ Via a dispatch rule (for example SIP or a room-creation trigger) ◦ Via the 
CreateAgentDispatchRequest
API 1. Is your agent server currently running and connected to LiveKit? ◦ If no agent server is available, jobs will not be accepted. 1. Are you seeing any error logs around the time a participant joins? ◦ Useful messages include things like “room connection was not established within 10 seconds” or other dispatch/failure logs. 1. Have there been recent changes to your agent code, dispatch rules, or deployment (configs, permissions, tokens)? 2. Could you share the relevant snippets: ◦ The agent entrypoint (where 
agent_name
might be set) ◦ How you create/join rooms and pass agent configuration (example token or 
RoomConfiguration
use) ◦ How you start the agent server (e.g., 
cli.run_app(server)
in Python) What I can check or suggest next (based on your answers) • If automatic dispatch is expected but an 
agent_name
is set, remove 
agent_name
or switch to explicit dispatch calls. • If dispatch is token-based, verify the token actually contains 
RoomConfiguration.agents
. • If dispatch rules are used, confirm the rule matched the room/participant event and that the rule targets an available agent. • If agent servers are down or overloaded, bring them back up or add capacity; jobs will not be accepted without available servers. • If you have logs showing connection timeout or similar, share them and I’ll point to the most likely remediation. If you prefer, paste the small code snippets and any recent error logs here and I’ll review them and give targeted steps to resolve it. Sources: How to Get Help from LiveKit | Agent dispatch | LiveKit Documentation | Server lifecycle | LiveKit Documentation | SIP Lifecycle Management Agent | LiveKit Documentation | Build a multilingual voice agent that automatically switches languages | Li...
e
@most-waitress-50456
t
Thanks — I see you mentioned @Olivia Herbert. Do you want me to direct the troubleshooting questions and next steps to Olivia specifically, or should I continue addressing them here? If you want us to proceed, please have Olivia (or whoever manages the agent setup) provide the details I asked for earlier: 1. Whether you use automatic or explicit dispatch (and whether an 
agent_name
is set in the agent entrypoint). 2. How agents are dispatched for the room (participation token via 
RoomConfiguration.agents
, dispatch rule, or 
CreateAgentDispatchRequest
API). 3. Whether the agent server is running and connected to LiveKit. 4. Any error logs around participant join times (e.g., connection timeouts or dispatch failures). 5. Recent changes to agent code, dispatch rules, or deployment. 6. Relevant code snippets: agent entrypoint, dispatch logic, and how you start the agent server (for example 
cli.run_app(server)
in Python). Once you share those details (or tell me to address Olivia directly), I’ll review them and provide targeted troubleshooting steps.
e
We know the workers are registering. And that our livekit tokens are being issued with the correct room options to spawn the given agent.
t
Thanks — I appreciate the extra detail that workers are registering and your tokens include the correct room options. That’s helpful. Next steps to narrow this down quickly 1. Confirm whether your agent is set up for explicit dispatch: ◦ Do you set an 
agent_name
in your 
@server.rtc_session
decorator or entrypoint? ◦ If 
agent_name
is set, automatic dispatch is disabled and the agent must be explicitly dispatched. 1. Confirm which explicit dispatch method you are using (pick one): ◦ Token-based dispatch: the token’s 
room_config
includes 
RoomAgentDispatch
with the same 
agent_name
. ◦ API dispatch: you call 
agent_dispatch.create_dispatch()
(or equivalent) with the correct 
agent_name
and room. ◦ Dispatch rule: a SIP/room-creation rule targets the correct 
agent_name
. 1. Quick checks you can run now ◦ Run the CLI agent status: 
lk agent status --id <agent-id>
— confirm agent shows as running/healthy. ◦ Verify the 
agent_name
in the agent entrypoint exactly matches the 
agent_name
in your dispatch config/token (string must match exactly). ◦ Inspect agent server logs at the moment a participant joins for messages about job rejection, timeouts, or connection attempts (e.g., “room connection was not established within 10 seconds”). 1. Please paste these snippets/logs (sanitized if needed) ◦ The 
@server.rtc_session
decorator or agent entrypoint where 
agent_name
might be set. ◦ The code or token payload showing the room’s 
room_config
/ 
RoomAgentDispatch
. ◦ Any agent/server logs from a join event (especially errors or timeouts). Why I’m asking these specifics • When workers are registered and tokens look correct, the most common cause is either: ◦ The agent is configured for explicit dispatch (so automatic spawn won’t happen), or ◦ A mismatch between 
agent_name
values (agent won’t accept jobs), or ◦ Agent servers are reachable but rejecting jobs due to startup/connect timeouts. If you share the three snippets/logs above, I’ll review them and give exact fixes—whether that’s removing 
agent_name
to enable automatic dispatch, updating your dispatch call/token to match the agent name, or addressing a connection/timeout issue in the agent server. Sources: Agent dispatch | LiveKit Documentation | Agents Framework | livekit/livekit_composite | DeepWiki
m
INFOlivekit.agentsregistered worker 104547.690 INFO … livekit.agents registered worker {"agent_name": "ds_agent_sauron", "id": "AW_oTguj9iyRd8r", "url": "wss://rtc.v23.rh0.ds.kinetic360.io", "region": "", "protocol": 16} lk agent status --id AW_oTguj9iyRd8r Using default project [ds_whisper] Using agent [AW_oTguj9iyRd8r] unable to list agents: failed to do request: Post "https://agents.v22.rh0.ds.kinetic360.io/twirp/livekit.CloudAgent/ListAgents": dial tcp: lookup agents.v22.rh0.ds.kinetic360.io on 127.0.0.5353 no such host
t
Thanks — this is very helpful. I can see two useful facts from what you shared: • Your worker successfully registered with LiveKit (agent_name "ds_agent_sauron") and reported a WSS URL. • The CLI call 
lk agent status
failed with a DNS resolution error when contacting the agents service endpoint: the client could not lookup agents.v22.rh0.ds.kinetic360.io. Next steps and things to check (quick, prioritized) 1. DNS / network resolution ◦ From the machine where you run 
lk agent status
, try resolving the hostname directly: ▪︎ Linux/macOS: 
nslookup <http://agents.v22.rh0.ds.kinetic360.io|agents.v22.rh0.ds.kinetic360.io>
or 
dig <http://agents.v22.rh0.ds.kinetic360.io|agents.v22.rh0.ds.kinetic360.io>
▪︎ Windows: 
nslookup <http://agents.v22.rh0.ds.kinetic360.io|agents.v22.rh0.ds.kinetic360.io>
◦ Confirm the command returns an IP. If it fails, this is a DNS issue (local resolver, corporate DNS, or firewall blocking DNS). 1. Host reachability ◦ If DNS resolves, test connectivity to the agents service host/port: ▪︎ 
curl -v <https://agents.v22.rh0.ds.kinetic360.io/>
(expect an HTTPS response or TLS handshake) ▪︎ Or 
telnet <http://agents.v22.rh0.ds.kinetic360.io|agents.v22.rh0.ds.kinetic360.io> 443
/ 
nc -vz <http://agents.v22.rh0.ds.kinetic360.io|agents.v22.rh0.ds.kinetic360.io> 443
◦ If these fail after DNS succeeds, a network route or firewall is blocking access. 1. Verify agent CLI configuration / project settings ◦ Confirm the CLI is pointed to the correct project/environment — you’re using default project [ds_whisper]. Ensure the agents host matches what your deployment expects (v22 vs v23 in logs). ◦ Note: the worker registered to a WSS URL with "v23" while the CLI is hitting an "agents.v22" hostname. Please confirm whether your environment should use v23 or v22 endpoints and that the CLI config and agent registration targets the same control plane version. 1. Compare versions/hosts for inconsistencies ◦ In the worker log: WSS URL shows rtc.v23.rh0.ds.kinetic360.io ◦ In the CLI error: agents.v22.rh0.ds.kinetic360.io ◦ If your environment migrated or has multiple control plane versions, the mismatch can cause the CLI to query an endpoint that isn’t reachable in your network. Confirm which agents host is correct for your deployment and update client configuration accordingly. 1. Agent server health vs. control plane reachability ◦ The worker registered, so the agent server likely reached the control plane at least once. The DNS failure from the machine running 
lk agent status
indicates the problem is between that machine and the control plane DNS/hosts, not necessarily the worker itself. ◦ If other machines or CI can reach the host, this could be a local DNS or VPN issue on your machine. What I need from you to continue • Results of the DNS resolution attempt (paste output of 
nslookup
or 
dig
) from the machine where the CLI runs. • Confirmation whether your expected control plane/agent host should be v22 or v23 (is there a known environment switch or deployment that moved from v22 → v23?). • Whether the CLI and the running workers are in the same network (same VPC, VPN, or on-prem environment). If you prefer, run these and paste the outputs here (sanitize any sensitive tokens). With the nslookup/curl outputs and confirmation of the expected host version (v22 vs v23), I’ll give exact next actions to restore agent dispatch (for example: update CLI/host config, fix DNS resolver, or route the hostname via /etc/hosts if appropriate).
e
We're self-deployed on premise. Should 
<http://agents.v23.rh0.ds.kinetic360.io|agents.v23.rh0.ds.kinetic360.io>
be a CNAME for 
<http://rtc.v23.rh0.ds.kinetic360.io|rtc.v23.rh0.ds.kinetic360.io>
, as it's the LiveKit server? Or should the agents host be separate?
t
Thanks — this is a great, focused question. I’ll answer it directly and give the practical checks to make next. Short answer • No: in a self‑deployed on‑prem environment, the agents hostname should not be a CNAME to the LiveKit server hostname unless the agents service and the LiveKit SFU are actually running on the same IP/instance. Agents and the LiveKit server are separate components and typically have distinct hostnames/IPs. Why (concise) • rtc.* (e.g., 
rtc.v23...
) is the LiveKit server (SFU) that handles WebRTC media and room management. • agents.* (e.g., 
agents.v23...
) represents the agents service or the endpoint for your agent worker pool. Agent workers connect outbound to the LiveKit server over WebSocket; they are a separate logical service. • Making 
agents.*
a CNAME to 
rtc.*
only makes sense if both services are served from the same host/IP and you intend requests to resolve to that same instance. Otherwise it hides the fact they are different services and can cause confusion or routing problems. Recommended setups • Agents and LiveKit on the same machine: ◦ You can point both hostnames to the same IP (A/AAAA) if you actually run both services there. A CNAME is not necessary. • Agents on separate worker pool (typical): ◦ 
agents.*
should resolve to the load-balancer or service IP for your agent pool (internal DNS or LB). ◦ 
rtc.*
should resolve to the LiveKit server(s)/LB that accept client and agent WebSocket connections. • Agents do not need a public DNS record if they only initiate outbound connections to LiveKit. Internal DNS or service names are fine. Practical checks to fix your issue 1. Verify what each host should point to in your architecture: ◦ Should agents be served from the same host as the SFU, or from a separate worker pool/LB? 1. Confirm agent env is correct: ◦ Agent workers must use LIVEKIT_URL pointing to the LiveKit server (e.g., 
<wss://rtc.v23>...
) plus correct API key/secret. 1. Make DNS consistent: ◦ If agents run separately, create an A/AAAA record (or internal DNS) for 
agents.v23...
pointing to the agent pool, not a CNAME to 
rtc.v23...
. ◦ If you must temporarily force resolution for debugging, you can map hostnames in /etc/hosts on a troubleshooting machine, but don’t rely on that for production. 1. Validate connectivity: ◦ From an agent host: confirm it can reach LiveKit WSS: 
openssl s_client -connect <http://rtc.v23.rh0.ds.kinetic360.io:443|rtc.v23.rh0.ds.kinetic360.io:443> -servername <http://rtc.v23.rh0.ds.kinetic360.io|rtc.v23.rh0.ds.kinetic360.io>
◦ From the CLI machine: ensure 
agents.v23...
resolves (nslookup/dig) and is reachable if your tooling needs to contact that endpoint. 1. Match control plane versions: ◦ In your logs you saw the worker register to 
rtc.v23...
while the CLI tried 
agents.v22...
— ensure your client tooling, agent registration, and DNS targets all reference the same version (v22 vs v23). If you want, tell me: • Whether your agent workers are on the same host as the LiveKit server or on a separate pool. • The output of a quick DNS check from the CLI machine (nslookup/dig for agents.v23...). I’ll walk through the exact DNS/hosts and connectivity changes to get dispatch working. Sources: Self-hosted deployments | LiveKit Documentation | Agents Framework | livekit/livekit_composite | DeepWiki | Building the all-in-one platform for voice AI agents
e
So how would you go about creating a debug/dev environment, when we're developing an agent locally?
t
Great — I’ll walk you through a practical, low-friction way to develop and debug an agent locally against your self‑hosted LiveKit. I’ll include quick checks and options so you can iterate fast and reproduce real behavior when needed. Overview — two common local dev modes • Console mode: fast local testing without a real LiveKit server. Good for developing voice pipeline logic and verifying agent behavior interactively. • Dev (local LiveKit) mode: connect your agent to a local or test LiveKit instance to validate actual room joins, media flows, and dispatch behavior. Step-by-step: quick local developer setup 1) Console mode — fastest iteration • Use console mode to run the agent inside your terminal and test audio/response logic without network complexity. • Start your agent in console mode (example CLI pattern used by agent toolchains): ◦ python/uv: 
uv run agent.py console
• Benefits: ◦ No LiveKit connectivity required. ◦ Immediate audio I/O in your terminal for debugging VAD, speech recognition, TTS, and state machine logic. • Use this for early unit/behavior testing before integrating with LiveKit. 2) Dev mode — integration testing with a LiveKit server • Run a local or test LiveKit server (or use your on‑prem dev cluster) and point your agent at it. • Set environment variables on the agent host: ◦ LIVEKIT_URL=wss://rtc.v23.rh0.ds.kinetic360.io (or your local server) ◦ LIVEKIT_API_KEY and LIVEKIT_API_SECRET • Start agent in dev mode so it registers with LiveKit and can be dispatched: ◦ python/uv: 
uv run agent.py dev
• Use this to validate actual job dispatch, participant joins, and end‑to‑end media behavior. 3) Logging and observability • Enable debug logging while developing: ◦ Start the agent with DEBUG log level or set logging in your code. ◦ Example: 
uv run agent.py start --log-level=DEBUG
• Inspect logs for connection handshakes, authentication, job acceptance/rejection, and WebSocket errors. • Tail both agent logs and LiveKit server logs during a test join to correlate events. 4) Prewarm heavy models and deterministic startup • Preload VAD / ASR models on agent start to avoid first‑request latency during tests. • Provide a small prewarm function that loads models into process memory once (mirrors production behavior). 5) Simulating inputs for reproducible tests • Scripted test clients: create a small test client or another agent that joins the same room and plays prerecorded audio or DTMF to trigger expected flows. • This produces repeatable test cases for unit and integration tests. 6) Handling dispatch behavior locally • If your agent entrypoint sets 
agent_name
, remember that automatic dispatch is disabled — you must explicitly dispatch (token-based or API). • For local dev: ◦ Issue test tokens containing 
room_config.agents
or call the dispatch API targeting your agent_name. ◦ Alternatively, remove 
agent_name
temporarily to allow automatic dispatch (only for local dev). 7) Networking tips for local on‑prem dev • If your LiveKit server and agent code run on different local hosts: ◦ Use internal DNS or /etc/hosts for quick name resolution only during dev. ◦ Ensure agent hosts can reach LiveKit WSS URL (test with 
openssl s_client -connect rtc...:443
). • Agents typically initiate outbound WSS connections to LiveKit — no public inbound to agent is required unless you expose HTTP endpoints for debugging. 8) Useful CLI/debug commands • Validate agent registration and health (where supported): ◦ 
lk agent status --id <agent-id>
(ensure it resolves to the correct agents host) • DNS/check connectivity: ◦ 
nslookup agents.v23...
or 
dig agents.v23...
◦ 
curl -v <https://agents.v23>...
(if applicable) ◦ 
openssl s_client -connect rtc.v23...:443 -servername rtc.v23...
9) Keep dev and prod configurations separate • Use environment-specific config (env files or docker-compose overrides). • Avoid using production API keys/secrets when developing locally. 10) Example local workflow 1. Start LiveKit server locally (or point to a dev cluster). 2. Start your agent in console mode to unit test logic. 3. Prewarm models in the agent process for faster runs. 4. Start agent in dev mode and confirm registration in LiveKit logs. 5. Create a test token that includes RoomConfiguration for agent dispatch and join the room with a test client that plays prerecorded audio. 6. Tail logs on both sides to confirm job acceptance and media flow. If you want, I can: • Give the exact command examples for your agent framework (paste your agent entrypoint or language and I’ll tailor commands). • Walk through creating a test token that will dispatch to your local agent (showing the minimal JSON/token payload). • Help you set up a small scripted client that plays test audio into a room for repeatable tests. Which would be most helpful next: example commands tailored to your agent code (Python vs Node), or a sample test token + script to auto-join a room and play a recording? Sources: Voice AI quickstart | LiveKit Documentation | Simple Call Answering Agent | LiveKit Documentation | Where to find your Agent Logs | LiveKit | examples/drive-thru | livekit/livekit_composite | DeepWiki | Job lifecycle | LiveKit Documentation
e
Would we have to edit our caddy config ?
t
Short answer: usually no — you don’t need to edit your Caddy config just to develop agents locally. Only change Caddy if you need the reverse proxy to expose or route traffic for a specific purpose in your dev flow. When you do NOT need to change Caddy • Running agent locally in console mode (no network): no proxy changes required. • Running agent locally but pointing it to your LiveKit server (agent initiates outbound WSS to LiveKit): no Caddy edits required — the agent makes outbound connections to the LiveKit WSS endpoint. • Using internal /etc/hosts or direct IPs to resolve names for quick local tests: no Caddy changes needed. When you MIGHT need to edit Caddy • You want to expose a local agent HTTP endpoint (health, metrics, or a webhook) under your normal domain (e.g., agents.v23...) so other components can reach it via the usual hostnames. • You want to proxy external test clients through the same public hostname as prod (so tokens and origins match exactly). • You need TLS termination or a specific hostname mapping for the agent pool that your tooling expects to resolve (e.g., if something calls agents.v23… and you want that to reach your local dev machine). What to change in Caddy if you do need it • Add a site block for the hostname you want to serve (or an internal dev hostname) and proxy to your local port: ◦ Example (conceptual): ▪︎ Host: agents.v23.rh0.ds.kinetic360.io ▪︎ Proxy to: http://127.0.0.1:8080 (or whichever local port your agent debug endpoint listens on) • Ensure the TLS settings match your dev needs: ◦ For local dev you can use 
tls internal
(Caddy’s local CA) or disable TLS and rely on an internal network if that’s safe for testing. • If your agent only needs outbound WSS to LiveKit, avoid exposing it publicly — prefer internal DNS or port-forwarding for safer testing. Alternative, lower‑impact options (prefer these for dev) • /etc/hosts mapping: map agents.v23… to localhost or your dev machine IP on just your dev computer for quick tests. • Use a local reverse proxy (Caddy/nginx) only on your dev machine — no cluster-wide change. • Use port-forwarding or SSH tunnel if you need another machine to reach your local agent. Quick checklist for your decision • Do you need inbound HTTP access to the local agent from other hosts? → Edit Caddy or add routing. • Do you only need the agent to connect outbound to LiveKit? → No Caddy change. • Do you want TLS with a public name during dev? → Configure Caddy to terminate TLS for that name and proxy to your local agent. If you want, tell me: • Do you need other hosts (e.g., browser clients, CI machines) to reach your local agent via agents.v23…? • Which port your local agent exposes (if any), and whether you prefer a /etc/hosts or proxy approach? I can then provide an exact Caddy site block example (or a safe /etc/hosts snippet) tailored to your environment.
Thanks — that log is exactly the kind of clue we need. The 401 with "invalid API key" means the LiveKit server rejected the agents service/CLI request because the API key used didn’t match the server’s configured credentials. What this indicates • Something (the CLI, agents service control endpoint, or an internal call) is sending an API key that the LiveKit server does not recognize. • That’s why the ListAgents twirp call fails with 401; the control plane is refusing the request before any agent dispatch logic runs. Practical steps to resolve (short checklist) 1) Confirm the server’s expected API key/secret • On the LiveKit server host, check how LIVEKIT_API_KEY and LIVEKIT_API_SECRET (or equivalent config) are set for that instance. • If you run LiveKit in Docker/Compose/K8s, check the environment or secret used by the livekit service. 2) Verify the client (agent/CLI) is using the same credentials • The client making the POST to /twirp/livekit.CloudAgent/ListAgents must use the matching API key/secret. • For the 
lk
CLI, ensure its configured project credentials match the LiveKit server values (check any config file, env vars, or CLI flags). • For agent workers, ensure their LIVEKIT_API_KEY/SECRET env vars are set to the same values the LiveKit server expects. 3) Check for multiple control-plane versions or environments • Your earlier logs showed v23 vs v22 differences. Confirm you’re pointing the CLI/agents at the same LiveKit control plane instance and that you aren’t mixing credentials from different environments. • If you have multiple LiveKit deployments, ensure each uses its own keys and the clients talk to the intended deployment. 4) Rotate / test keys safely • If you’re uncertain which secret is active, temporarily set a known key/secret pair on the LiveKit server and update your local agent/CLI to use that same pair for a quick sanity test. • After confirmation, restore intended secrets or adopt the new pair across components. 5) Inspect server logs for auth failures • Tail LiveKit server logs while making the failing request to confirm it’s the same 401 and to see if additional context appears. 6) Common gotchas • Trailing whitespace or accidental quoting in env vars (e.g., quotes included in value) can make keys mismatch. • Using base64 or encoded secrets where the server expects raw secret (or vice versa). • Old credentials cached in a CLI config file — check ~/.config or where 
lk
stores project credentials. If you’d like, I can help validate the exact values without exposing secrets here: • Tell me where the livekit server reads its API key (env var, k8s secret, docker-compose file) and where the client (CLI or agent) reads its key. • Paste sanitized snippets of how you set LIVEKIT_API_KEY/SECRET (replace actual secrets with placeholders) and the command you use to run the CLI/agent. If you prefer a quick test now 1. On the LiveKit server host, temporarily set: ◦ LIVEKIT_API_KEY=testkey ◦ LIVEKIT_API_SECRET=testsecret 1. Restart the LiveKit process. 2. On your CLI/agent machine export the same values and retry 
lk agent status
or re-register an agent. If the call succeeds, it confirms a credentials mismatch; then you can reconcile config across services. Tell me which option you want (check specific envs, run the temporary test, or paste sanitized config) and I’ll walk through the exact commands.
e
We added a caddy entry so that the livekit server also answers for the agents requests.
t
Thanks — that’s a helpful change and it explains the path your requests now take. Since you added a Caddy entry so the LiveKit server also answers agents requests, the most likely cause of the 401 “invalid API key” is the reverse proxy not forwarding the authentication headers (or TLS/host mismatches) to the LiveKit control plane. What to check and fix (practical, ordered) 1) Ensure Authorization header is preserved • The LiveKit control plane expects the API key/secret (or the Authorization header built from them) on incoming requests. If Caddy strips or rewrites that header the server will return 401. • Make sure your proxy config does not remove Authorization. In Caddy this is controlled by proxy header handling — explicitly forward Authorization. 2) Ensure Host/TLS/SNI align with LiveKit config • If LiveKit validates the Host header or TLS SNI, proxying to a different Host or terminating TLS at Caddy can change the incoming Host/SNI seen by LiveKit. Confirm LiveKit accepts the proxied Host or update Caddy to preserve it (or set the Host header explicitly when proxying). • If you terminate TLS at Caddy and speak HTTP to the backend, LiveKit must be okay with the HTTP Host header you provide. 3) Caddy proxy example (conceptual) • If you want Caddy to proxy agents.v23.rh0.ds.kinetic360.io to your LiveKit control-plane backend and preserve auth/host, add explicit header passthrough. Example site block (conceptual — adapt to your Caddyfile syntax and internal backend address): ◦ Ensure Authorization is forwarded: ▪︎ header_up Authorization {>Authorization} ◦ Preserve Host if needed: ▪︎ header_up Host {host} ◦ Example (for a reverse_proxy directive): ▪︎ reverse_proxy <livekit-backend-host>:<port> { header_up Authorization {>Authorization} header_up Host {host} } 4) Validate behavior end-to-end • After updating Caddy, test with the same failing CLI/agent command and tail LiveKit logs. • Check that the request now reaches LiveKit and that the Authorization header is present. You should no longer receive 401. • Useful quick checks: ◦ From the proxy host, do a curl that includes the same Authorization header the client would send and verify the backend sees it: ▪︎ curl -v -H "Authorization: Bearer <sanitized-token>" https://agents.v23.rh0.ds.kinetic360.io/twirp/livekit.CloudAgent/ListAgents ◦ Tail LiveKit server logs during the request to confirm it accepted the auth. 5) Common gotchas • Casing/spaces/format: ensure Authorization header format exactly matches expected (no accidental quoting or extra whitespace). • Multiple proxies: if requests traverse more than one proxy, ensure none of them strip Authorization. • Cookie or header rewriting: some proxy rules strip or cleanse headers for security — explicitly preserve Authorization for agent endpoints. • If you terminate TLS at Caddy, ensure LiveKit accepts the proxied host or use 
header_up Host {host}
so the backend sees the expected Host header. If you want, I can: • Draft an exact Caddy site block for your environment if you paste the current Caddyfile snippet (remove any secrets). • Give an exact curl command to test header passthrough that you can run locally. • Walk through interpreting LiveKit logs to confirm auth success. Which would help most right now: a concrete Caddyfile example tailored to your setup, or the verification curl command to run immediately?
m
lk agent status --id AW_6xjpTa9dU4cz Using default project [ds_agent_sauron] Using agent [AW_6xjpTa9dU4cz] unable to list agents: Error from intermediary with HTTP status code 404 "Not Found" What does it mean?
t
Thanks — that 404 is a useful clue. I’ll explain what it means in this context and give targeted checks to resolve it. What a 404 means here • The proxy (Caddy) forwarded your request but the backend returned HTTP 404 Not Found. In other words, the intermediary accepted the connection, but the service it reached did not recognize the requested path (/twirp/livekit.CloudAgent/ListAgents) or the host routing didn’t map to an agents endpoint that implements that API. • This is different from the earlier 401 (auth) — it shows the request now reaches some HTTP endpoint, but not the correct agents control API. Likely causes (ordered by probability) • Caddy is routing to the wrong backend service (e.g., routing to the SFU’s public HTTP handler which doesn’t expose the agents twirp endpoint). • The Host header or path was rewritten by the proxy so the backend didn’t match the twirp route. • The backend service is a different version/environment that doesn’t expose the CloudAgent service at that path (v22 vs v23 mismatch). • The agents service is not running on the backend you hit, or the endpoint is mounted on a different path. Concrete checks to run now 1. Confirm which backend Caddy is proxying to for the agents hostname ◦ Inspect the Caddyfile site block you added for agents.v23... and verify the reverse_proxy target address (IP:port or upstream name). 1. Test the backend directly (bypass Caddy) ◦ From the machine that runs Caddy (or where you can reach the upstream directly), curl the backend service address that Caddy proxies to and hit the same path: ▪︎ curl -v http//&lt;backend ip&gt;<port>/twirp/livekit.CloudAgent/ListAgents ◦ If that returns 404, the backend itself doesn’t implement that route. ◦ If it succeeds, compare response headers with the proxied request to find what Caddy might be changing. 1. Verify Host/header passthrough in Caddy ◦ Ensure Caddy preserves or sets the correct Host and Authorization headers for the proxied request: ▪︎ header_up Host {host} ▪︎ header_up Authorization {>Authorization} ◦ If Caddy uses a different Host header when proxying, the backend may treat it as a different virtual host and not expose the twirp endpoint. 1. Confirm the expected path and service version ◦ Double-check the correct path for the agents API in your deployment. The CLI calls the twirp path `/twirp/livekit.CloudAgent/ListAgents`; ensure your backend is the agents control plane and not the SFU HTTP endpoint. ◦ Ensure the CLI, Caddy, and agent backends all reference the same control-plane version (v23 vs v22). 1. Check backend service availability and logs ◦ Tail logs on the service Caddy proxies to and look for the incoming request and how it’s handled. The logs should show the incoming path and why it returned 404. ◦ If no logs, the request isn’t reaching the intended agents service. Quick tests you can paste results from • Your Caddyfile snippet for the agents site block (remove secrets). • Output of: ◦ curl -v https://agents.v23.rh0.ds.kinetic360.io/twirp/livekit.CloudAgent/ListAgents ◦ curl -v http//&lt;backend ip&gt;<port>/twirp/livekit.CloudAgent/ListAgents (direct to upstream) • The LiveKit/agents backend logs around the request time. Next steps I can do with those results • If Caddy is routing to the SFU HTTP endpoint, I’ll show an edited site block that routes the agents hostname to the correct agents control port. • If headers are being rewritten, I’ll provide the exact Caddy config lines to preserve Host and Authorization. • If the backend lacks the agents endpoint, I’ll point out the expected service to run (the agent control plane) and how to target it. If you want, paste your Caddy site block for the agents host and one curl result (proxied and direct). I’ll mark exactly what to change so the agents API returns the ListAgents response instead of 404.
e
We fixed the token issue, but now we get:
Copy code
lk agent status --id AW_6xjpTa9dU4cz

Using default project [ds_agent_sauron]

Using agent [AW_6xjpTa9dU4cz]

unable to list agents: Error from intermediary with HTTP status code 404 "Not Found"
 
well. we're closer.
t
Great — you’re right, that 404 means you’re very close. It shows the request is reaching something (your proxy) but not the correct agents control API. Let’s finish the last checks to get the agents CLI talking to the right service. What the 404 tells us • Your request now passes the proxy, but the backend handling it does not expose the Twirp agents route 
/twirp/livekit.CloudAgent/ListAgents
. Either Caddy is routing to the wrong upstream, or headers/host/path are being changed so the backend doesn’t match the API. Please run these quick checks (ordered to be fastest/most likely) 1) Confirm what Caddy is proxying for the agents host • Inspect the Caddyfile site block you added for agents.v23.rh0.ds.kinetic360.io and paste the block here (remove any secrets). I’ll check the upstream address and header passthrough lines. 2) Curl the proxied endpoint • From any machine that hits the proxy, run: curl -v https://agents.v23.rh0.ds.kinetic360.io/twirp/livekit.CloudAgent/ListAgents • Paste the verbose output (remove Authorization value). That shows what the proxy returns and any redirect/host info. 3) Curl the backend directly (bypass Caddy) • Find the upstream backend IP:PORT you configured in Caddy and run: curl -v http//&lt;backend ip&gt;<port>/twirp/livekit.CloudAgent/ListAgents • If that returns 404 too, the actual backend doesn’t expose the agents API at that path (wrong service or wrong path). • If that succeeds but the proxied request fails, the proxy is altering the request (host/header/path). 4) Verify Host and header passthrough in Caddy • Ensure Caddy forwards the Host and Authorization headers. In reverse_proxy blocks, you should include: ◦ header_up Host {host} ◦ header_up Authorization {>Authorization} • If you terminate TLS at Caddy and proxy HTTP to the backend, be sure the Host header equals what the backend expects. 5) Confirm the backend is the agents control plane • The backend Caddy should route to must be the agents control service (the one that implements livekit.CloudAgent). If you accidentally proxied to the SFU/public HTTP handler, that will return 404 for the Twirp agent route. • Double-check the upstream port you set in Caddy is the agents control API port, not the SFU port. 6) Check backend logs when you curl • Tail logs on the upstream service while making the request. If the backend logs show the incoming path and returns 404, it’s the wrong service. If no log appears, the request isn’t reaching that backend at all. If you paste two things I’ll give exact edits: • Your agents site block from the Caddyfile (sanitized). • The outputs of the two curl commands (proxied and direct). With those I’ll tell you whether to: • Change Caddy to point to the correct upstream, • Preserve Host/Authorization headers, or • Update the upstream service to expose the Twirp agents API on the expected path. You’re almost there — paste the Caddy snippet and curl outputs and I’ll provide the exact config changes.
e
Copy code
GNU nano 7.2                                                                       caddy.yaml                                                                             S  
logging:
  logs:
    default:
      level: INFO
storage:
  "module": "file_system"
  "root": "/data"
apps:
  tls:
    certificates:
       load_files:
        - certificate: "/etc/caddy/certs/fullchain.pem"
          key: "/etc/caddy/certs/privkey.pem"
#      automate:
#        - <http://video.rh0.ds.kinetic360.io|video.rh0.ds.kinetic360.io>
#        - <http://turn.rh0.ds.kinetic360.io|turn.rh0.ds.kinetic360.io>
  layer4:
    servers:
      main:
        listen: [":443"]
        routes:
          - match:
            - tls:
                sni:
                  - "<http://turn.v23.rh0.ds.kinetic360.io|turn.v23.rh0.ds.kinetic360.io>"
            handle:
              - handler: tls
              - handler: proxy
                upstreams:
                  - dial: ["localhost:5349"]
          - match:
            - tls:
                sni:
                  - "<http://rtc.v23.rh0.ds.kinetic360.io|rtc.v23.rh0.ds.kinetic360.io>"
                  - "<http://agents.v23.rh0.ds.kinetic360.io|agents.v23.rh0.ds.kinetic360.io>"
            handle:
              - handler: tls
                connection_policies:
                  - alpn: ["http/1.1"]
              - handler: proxy
                upstreams:
                  - dial: ["localhost:7880"]
          - match:
            - tls:
                sni:
                  - "<http://bridge.v23.rh0.ds.kinetic360.io|bridge.v23.rh0.ds.kinetic360.io>"
            handle:
              - handler: tls
                connection_policies:
                  - alpn: ["http/1.1"]
              - handler: proxy
                upstreams:
                  - dial: ["localhost:8099"]
What do you think?
t
Thanks — I appreciate the context and the progress you’ve made. You’re definitely close, and your Caddy config plus the logs you shared give enough signal to recommend the exact next steps. Summary of current state • Workers are able to register (good). • You fixed the token/API key problem (great). • 
lk agent status
now reaches your proxy but returns 404 — meaning the proxy is forwarding requests, but the backend receiving them does not expose the Twirp agents API path (/twirp/livekit.CloudAgent/...). In short: proxy routing or headers are not landing on the correct LiveKit control API endpoint. Why the 404 happens (concise) • Caddy is routing SNI for both rtc.* and agents.* to localhost:7880 (LiveKit). That part is fine. • A 404 indicates the service that Caddy ultimately forwarded to either: ◦ isn’t the control-plane component that exposes the CloudAgent Twirp endpoints, or ◦ is receiving a different Host/path header than it expects, so it doesn’t match the Twirp route. Concrete next actions — fast checks and fixes 1. Confirm backend target is the control-plane process ◦ Ensure the process listening on localhost:7880 is the LiveKit control plane that exposes Twirp agent routes (not just the SFU public HTTP or a different service). ◦ On the machine running Caddy: ▪︎ curl the backend directly: ▪︎ curl -v http://localhost:7880/twirp/livekit.CloudAgent/ListAgents ▪︎ If that returns 404, the service on :7880 is not the agent control API (wrong upstream). If it succeeds, continue to step 2. 1. Preserve Host and headers in the proxy ◦ Make sure Caddy forwards Host and Authorization headers so the backend sees the expected hostname and auth: ▪︎ header_up Host {host} ▪︎ header_up Authorization {>Authorization} ◦ If you terminate TLS in Caddy and proxy HTTP to LiveKit, ensure the Host header equals what LiveKit expects (rtc.v23... or agents.v23... as appropriate). 1. Ensure Twirp paths are routed correctly ◦ Twirp endpoints are path-based under /twirp. If your Caddy rules do any path rewriting or subrouting, ensure /twirp/* requests reach the same upstream without being altered. ◦ If you want explicit path routing, use a subroute that proxies /twirp/* to localhost:7880. 1. ALPN / HTTP version (minor but worth validating) ◦ Allow both 
h2
and 
http/1.1
in connection_policies so HTTP/2/gRPC-style traffic can reach the backend: ▪︎ connection_policies: • alpn: ["h2","http/1.1"] 1. Confirm which hostname LiveKit expects for agent API ◦ The agent control API may expect a specific Host header. If your LiveKit control plane was configured for a name (e.g., rtc.v23...), either proxy with that Host or configure LiveKit to accept the agents.v23 host as well. 1. Quick verification steps to run and paste results ◦ From the Caddy host: ▪︎ curl -v http://localhost:7880/twirp/livekit.CloudAgent/ListAgents ◦ From the client machine (the one that runs 
lk agent status
): ▪︎ curl -v https://agents.v23.rh0.ds.kinetic360.io/twirp/livekit.CloudAgent/ListAgents ◦ Paste both outputs (sanitize Authorization header) and your current agents site block if you want me to review it again. If direct backend curl succeeds but proxied curl returns 404 • The issue is Caddy rewriting Host or path — add header_up lines and ensure no path rewrite happens. If direct backend curl returns 404 • The upstream bound to :7880 is not the control-plane service exposing Twirp. Update Caddy to point agents.v23.* to the correct upstream (the process/port that implements livekit.CloudAgent). Offer to help with exact edits • If you paste the (sanitized) Caddy site block for rtc/agents and the two curl outputs, I’ll give the exact Caddy config snippet you need (header_up lines or corrected upstream address) and a curl command to confirm success. You’re nearly there — once the proxy routes /twirp requests (with Host/Authorization preserved) to the control-plane endpoint, 
lk agent status
will return the agent list and agents will accept jobs. Want me to draft the exact Caddy snippet to add header passthrough and an optional /twirp subroute?
e
Copy code
# curl -v <http://localhost:7880/twirp/livekit.CloudAgent/ListAgents>
* Host localhost:7880 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:7880...
* Connected to localhost (::1) port 7880
> GET /twirp/livekit.CloudAgent/ListAgents HTTP/1.1
> Host: localhost:7880
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Vary: Origin
< X-Content-Type-Options: nosniff
< Date: Wed, 25 Feb 2026 17:29:00 GMT
< Content-Length: 19
< 
404 page not found
* Connection #0 to host localhost left intact
How do you normally deploy local ai agents? What about twirp?
t
Thanks — this is very helpful context. You ran the direct curl to localhost:7880 and got a 404; that confirms the service on :7880 does not expose the CloudAgent Twirp endpoints. I’ll summarize what that means and give a clear, practical plan for local agent development and what to change next. What the localhost 404 means • The process listening on localhost:7880 is not serving the CloudAgent Twirp API (/twirp/livekit.CloudAgent/...). • In self‑hosted LiveKit, the CloudAgent Twirp endpoints (the managed Cloud control-plane agents API) are not present by default. Your local LiveKit server exposes room/participant/egress/ingress Twirp endpoints, but not the CloudAgent control API that the LiveKit Cloud product provides. Because that endpoint is missing, requests to /twirp/livekit.CloudAgent/* return 404. Implications • The 
lk agent status
or any client expecting CloudAgent Twirp routes will return 404 against a self‑hosted control plane that doesn’t implement that service. • Agent worker registration and job dispatch in self‑hosted setups is handled differently: agents connect to the LiveKit server as participants over WSS (LIVEKIT_URL), and dispatching is via token room config, explicit dispatch APIs your deployment provides, or an internal agent-dispatch component — not the CloudAgent Twirp endpoint you queried. Practical next steps (two parallel tracks) A) If your goal is local development of agents (recommended) 1. Run and test agents locally using the developer modes: ◦ Console mode (no LiveKit required) for voice pipeline and logic: ▪︎ Example: 
uv run agent.py console
or run your agent entrypoint in console/dev variant. ◦ Dev mode (connects to your self‑hosted LiveKit) to test real room joins: ▪︎ Set env: ▪︎ 
LIVEKIT_URL=<wss://rtc.v23.rh0.ds.kinetic360.io>
(or 
<ws://localhost:7880>
for local HTTP) ▪︎ 
LIVEKIT_API_KEY
/ 
LIVEKIT_API_SECRET
▪︎ Start agent: 
uv run agent.py dev
(or your framework’s dev command). 1. Ensure your agent entrypoint does not set 
agent_name
if you want automatic dispatch: ◦ Remove 
agent_name
from 
@server.rtc_session()
to allow auto-dispatch. ◦ If you do set 
agent_name
, you must explicitly dispatch via token or API. 1. Test agent registration and job handling: ◦ Start agent, then create a room (or join a room) with a token that includes 
RoomConfiguration.agents
or call your dispatch API to assign the agent to the room. ◦ Tail agent logs and LiveKit server logs to confirm the agent joins as a participant and accepts jobs. 1. Use scripted test clients for repeatable flows: ◦ A small client that joins the room and plays prerecorded audio to trigger the agent is useful for automated tests. B) If your intention was to make CloudAgent Twirp API calls (e.g., 
lk agent status
) 1. Accept that CloudAgent Twirp endpoints are not provided by the default self‑hosted LiveKit server. ◦ You will continue to get 404 trying to reach 
/twirp/livekit.CloudAgent/*
on your self‑hosted server. 1. Options if you need 
lk
CLI agent-listing functionality: ◦ Use the CLI features that talk directly to your environment’s agent control endpoint if you have a custom agent control service. ◦ Or, run the 
lk
CLI against the cloud-managed control plane (if you use LiveKit Cloud) — otherwise, adapt your tooling to the self‑hosted control interfaces (tokens, room-config dispatch, custom admin endpoints you provide). C) Quick Caddy and connectivity notes (to avoid future proxy issues) • You already proxy rtc.* and agents.* to localhost:7880. That’s fine; the issue here is the upstream service itself doesn’t implement CloudAgent Twirp. • If you ever proxy a control API, ensure Caddy preserves headers: ◦ forward Host and Authorization: ▪︎ header_up Host {host} ▪︎ header_up Authorization {>Authorization} • Allow both HTTP/2 and HTTP/1.1 in ALPN for gRPC/Twirp traffic: ◦ connection_policies: alpn: ["h2","http/1.1"] Examples and commands you can run now 1. Run agent locally (dev mode) ◦ Export env: ▪︎ export LIVEKIT_URL=wss://rtc.v23.rh0.ds.kinetic360.io ▪︎ export LIVEKIT_API_KEY=<your_key> ▪︎ export LIVEKIT_API_SECRET=<your_secret> ◦ Start agent: 
uv run agent.py dev
(or your agent start command) 1. Create a token that includes agent dispatch (example pseudocode) ◦ Create a room token containing RoomConfiguration.agents that references your agent (or omit agent_name for auto‑dispatch). 1. Tail logs: ◦ Agent logs: watch for “registered worker” and “job received” ◦ LiveKit logs: watch for participant/agent joins and room events If you want, I can do one of the following next (pick one) • Show the exact minimal agent entrypoint and commands (Python or Node) to run locally in console/dev mode. • Provide a sample token payload that will dispatch an agent into a room for testing. • Help adapt your 
lk
CLI usage to work with your self‑hosted environment (i.e., explain what commands will work and which require the Cloud-managed Twirp endpoints). Which would you like me to provide next? Sources: Server-side | livekit/livekit_composite | DeepWiki | An open source stack for real-time multimodal AI | Agent dispatch | LiveKit Documentation | server-sdk-go | livekit/livekit_composite | DeepWiki | Examples | livekit/livekit_composite | DeepWiki
e
We are using tokens with agent configs.
t
Thanks — great to know you’re already using tokens with agent configs. That’s the right path for self‑hosted deployments. I’ll summarize what’s happening, show a concrete token example you can use, and give a short checklist + verification steps to make sure the agent actually launches when a participant joins. Quick summary of how token-based agent dispatch works (self-hosted) • Your agent workers connect to the LiveKit server as participants over WSS using LIVEKIT_URL / API key/secret. They register and stay available to accept jobs. • To cause an agent to join a room automatically when someone joins, include an agent dispatch entry in the room configuration embedded into the room-join token. LiveKit will use that room_config to dispatch an agent to the room. • You do NOT need CloudAgent Twirp endpoints for token-based dispatch; the CloudAgent API is a Cloud product feature. Token dispatch works with your self-hosted server as long as agents are registered and reachable by the LiveKit server. Concrete example — create a token that requests an agent (Replace placeholders with your actual values) • Example payload / pseudocode to produce a room token that includes a RoomAgentDispatch:
Copy code
# pseudocode (adjust to your SDK)
from livekit import AccessToken, VideoGrant, RoomConfiguration, RoomAgentDispatch

token = AccessToken(api_key="YOUR_API_KEY", api_secret="YOUR_API_SECRET")
token.add_identity("participant-1")

# include room join grant (if your SDK uses one)
token.add_grant(VideoGrant(room_join=True, room="test-room"))

# attach RoomConfiguration with agent dispatch
room_agent = RoomAgentDispatch(
    agent_name="ds_agent_sauron",   # optional: if omitted, auto-dispatch behavior differs
    metadata='{"role":"assistant"}'
)

room_config = RoomConfiguration(agents=[room_agent])
token.set_room_configuration(room_config)

jwt = token.to_jwt()
Important notes on the fields: • `agent_name`: If you set an 
agent_name
in your agent entrypoint code, that agent will only accept jobs targeted to that name — this is explicit dispatch. If you want automatic, ensure agent entrypoint does NOT set a fixed 
agent_name
(remove it from 
@server.rtc_session()
), and then the token’s room_config can simply request an agent kind rather than a specific name. • `metadata`: optional JSON string passed to the agent job (useful to pass user/context). Checklist — why an agent might still not launch 1. Agent worker registration ◦ Confirm agent worker logged “registered worker” and shows healthy. Worker must be running and connected to the LiveKit server. 1. Correct LIVEKIT_URL and credentials ◦ Worker env: LIVEKIT_URL must point to your LiveKit server (wss://rtc.v23... or ws://localhost:7880 for local dev). ◦ LIVEKIT_API_KEY / SECRET on worker must match server keys. 1. Token actually contains room_config.agents ◦ Inspect the token payload (decode JWT) and confirm 
room_config
includes the agent entry you expect. 1. agent_name mismatch ◦ If the worker sets 
agent_name
in code, the token must target that same name. If worker has no 
agent_name
, tokens that request agents without names should allow auto-dispatch. 1. Network / proxy ◦ LiveKit server must be able to dispatch to registered workers (they connect outbound). Your Caddy proxy should preserve Host and Authorization when proxying API requests and not rewrite the /twirp path if you use any admin APIs. Verification steps you can run now (fast) 1. Decode the token you create and confirm it contains 
room_config
-> 
agents
. You can paste the token (sanitized) or decode at jwt.io locally. 2. Start your agent worker in verbose mode and tail logs. 3. Join the room as a normal participant with the token you created. 4. Watch logs: ◦ LiveKit server logs should show a dispatch/job creation event. ◦ Agent logs should show job acceptance and “joined room” events. 1. If no agent appears: ◦ Check agent logs for job rejections, timeouts, or auth errors. ◦ Ensure token’s 
room
value matches the room the participant joined. If you want, I can do one of these next: • Draft a concrete token-generation snippet in the exact language/SDK you use (Python/Node/Go), showing the minimal fields that must be present. • Review the token contents if you paste the decoded JWT (sanitize values). • Provide the exact agent entrypoint example to toggle between auto vs explicit dispatch (showing where to remove or set 
agent_name
). Which would be most helpful: a ready-to-run Python token snippet, or a quick review of a decoded token you have?
e
Copy code
def generate_token(self, room_name: str, identity: str, is_admin: bool = False) -> str:
        """Generate a LiveKit token for a participant to join a room.
        Args:
            room_name: The name of the room to join
            identity: User identity
            is_admin: Whether to grant admin privileges
        Returns:
            JWT token string
        """
        identity = identity or self.identity

        token = (
            api.AccessToken(self.lk_key, self.lk_secret).with_identity(identity).with_name(identity)
        )
        grants = {
            'room_join': True,
            'room': room_name,
            'can_publish': True,
            'can_subscribe': True,
            'can_publish_data': True,  # Grant permission to publish data messages
        }
        if is_admin:
            grants.update(
                {
                    'room_record': True,
                    'room_admin': True,
                    'ingress_admin': True,
                }
            )
            # Admin tokens are often used by services, so setting kind to egress might be appropriate.
            # However, for general participant admin, it might not be strictly necessary.
            # Keeping it as per original logic.
            token = token.with_kind('egress')

        token = token.with_ttl(pendulum.Duration(days=7)).with_grants(api.VideoGrants(**grants))
        token.with_room_config(
            api.RoomConfiguration(
                agents=[
                    api.RoomAgentDispatch(agent_name="ds_agent_sauron", metadata=json.dumps({"username": identity})),
                ]
            )
        )
        return token.to_jwt()
t
Thanks — this is exactly the right place to focus. You’re already using tokens with agent configs; the issue was that your token-building call to 
with_room_config()
wasn’t being reassigned, so the room_config (with the agent dispatch entry) was never included in the final token. That explains why agents weren’t launching. What I looked at • Your generate_token() builds a token, then calls `token.with_room_config(...) 
but doesn’t reassign the return value.
with_room_config()` returns a new AccessToken instance, so the room_config is being dropped before 
to_jwt()
is called. What to change (concrete fix) • Chain or reassign the call so the RoomConfiguration is included in the final token. Corrected version (minimal, ready to use):
Copy code
def generate_token(self, room_name: str, identity: str, is_admin: bool = False) -> str:
    identity = identity or self.identity

    token = (
        api.AccessToken(self.lk_key, self.lk_secret)
        .with_identity(identity)
        .with_name(identity)
        .with_ttl(pendulum.Duration(days=7))
        .with_grants(api.VideoGrants(
            room_join=True,
            room=room_name,
            can_publish=True,
            can_subscribe=True,
            can_publish_data=True,
        ))
        .with_room_config(
            api.RoomConfiguration(
                agents=[
                    api.RoomAgentDispatch(
                        agent_name="ds_agent_sauron",
                        metadata=json.dumps({"username": identity}),
                    ),
                ]
            )
        )
    )

    # If you set .with_kind(...) for admin earlier, include that in the chain instead.
    return token.to_jwt()
Why this fixes it • Each 
with_*
method returns a new token object. By chaining (or assigning the return), the RoomConfiguration becomes part of the token payload and will be honored by the LiveKit server at join time. Quick verification steps 1. Decode the token you generate (locally, e.g., jwt.io or your own decoder) and confirm the payload contains: ◦ A 
room_config
field with 
agents
containing an entry for 
"agent_name": "ds_agent_sauron"
and the metadata you set. 1. Confirm agent naming and dispatch mode: ◦ If your agent code sets 
agent_name="ds_agent_sauron"
in the entrypoint, that agent will only accept jobs targeted to that name (explicit dispatch). That matches your token approach. ◦ If you want automatic dispatch (agent spawn for rooms without naming), remove 
agent_name
from the agent entrypoint. 1. Tail logs during a test: ◦ Start the agent worker (verbose DEBUG logs). ◦ Join the room with the corrected token. ◦ Watch LiveKit logs for a dispatch/job creation event and the agent logs for job acceptance / "joined room" messages. Troubleshooting notes if it still doesn’t spawn • Agent registration: confirm the agent process logs “registered worker” and is healthy. • Credentials: ensure agent process uses the same LIVEKIT_API_KEY/SECRET that the LiveKit server expects. • Network/proxy: ensure LiveKit can reach the agent (agents typically connect outbound to the LiveKit server; no public inbound is required). • Token room name: ensure the room name in the token matches the room the participant actually joins. If you want, I can: • Review a decoded token (paste the payload, sanitized) to confirm room_config is present. • Provide a one-off test token generator for your environment. • Show the exact agent entrypoint code snippet (Python) to confirm whether 
agent_name
is set (and how to switch to auto-dispatch). Which would you like me to do next? Sources: Agent dispatch | LiveKit Documentation
e
Copy code
ic| f'Generated token: {token}': 'Generated token: <livekit.api.access_token.AccessToken object at 0x7768a3ef4c50>'
ic| '--- Token Claims ---'
ic| f"Identity: {claims.identity}": 'Identity: <mailto:video@api.v23.rh0.ds.kinetic360.io|video@api.v23.rh0.ds.kinetic360.io>'
ic| f"Grants: {claims.video}": 'Grants: VideoGrants(room_create=None, room_list=None, room_record=None, room_admin=None, room_join=True, room='kinetic', destination_room=None, can_publish=True, can_subscribe=True, can_publish_data=True, can_publish_sources=None, can_update_own_metadata=None, ingress_admin=None, hidden=None, recorder=None, agent=None)'
ic| "\n--- Agent Configuration ---": '''
                                      --- Agent Configuration ---'''
ic| f"Agent Name: {agent.agent_name}": 'Agent Name: ds_agent_sauron'
ic| f"Agent Metadata: {agent.metadata}": 'Agent Metadata: {"username": "<mailto:video@api.v23.rh0.ds.kinetic360.io|video@api.v23.rh0.ds.kinetic360.io>"}'
t
Thanks — this is great evidence. Your decoded claims show the token does include the room join grant and a room agent entry with the exact agent name and metadata, so the token is now being generated correctly. What your output shows (good) • Identity: video@api.v23.rh0.ds.kinetic360.io • Video grants: room_join=True and publish/subscribe permissions present. • Agent config: agent_name = "ds_agent_sauron" and metadata = {"username": "..."}. Because the token is correct, we can now focus on the remaining places where things commonly break. Here’s a short checklist of the next verification steps and exactly what to look for in logs so we can pinpoint the final issue quickly. Checklist — what to verify now 1) Agent worker registration (must be running and healthy) • Confirm agent process logs show: "registered worker" and that the agent registers with the same name: ◦ Example log entry you should see: registered worker {"agent_name":"ds_agent_sauron", "id":"AW_..."} • If you don’t see that, start/inspect the agent process and its env (LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET). 2) Agent entrypoint name and dispatch mode • If your agent code sets 
@server.rtc_session(agent_name="ds_agent_sauron")
, explicit dispatch is expected — which matches your token. That is OK. • If instead you want auto-dispatch (no specific name), remove the agent_name from the decorator and re-run the agent. 3) Confirm the participant actually joins the same room name in the token • Your token shows room = "kinetic". Make sure the client that joins uses that same room name. A mismatch prevents dispatch. 4) Watch LiveKit server logs during join • Tail the LiveKit server logs and look for a dispatch or job creation log when the participant joins. Useful messages: ◦ job creation / job assigned / dispatch created ◦ Any errors about “no available workers” or “job rejected” • If you see no dispatch log, LiveKit didn’t create a job — check token room and server config. 5) Watch the agent logs during the join • Tail agent logs while a participant joins. Look for: ◦ job received / job accepted ◦ room connection established or connection timeout messages ◦ authentication or permission errors • If you see “room connection was not established within 10 seconds” or similar, that indicates agent failed to connect to the room in time (resource/capacity/network issue). 6) Verify LIVEKIT_URL and connectivity for the agent • The agent must connect to the LiveKit server WSS endpoint. Confirm agent env uses the correct URL: ◦ For your setup: LIVEKIT_URL should be wss://rtc.v23.rh0.ds.kinetic360.io (or ws://localhost:7880 for local testing) • From agent host, test connectivity: ◦ openssl s_client -connect rtc.v23.rh0.ds.kinetic360.io:443 -servername rtc.v23.rh0.ds.kinetic360.io 7) Confirm proxy/host alignment (if applicable) • You previously proxied agents and rtc via Caddy. Ensure the LiveKit server accepts the Host header the proxy sends. If the proxy rewrites Host or strips Authorization, agent registration or dispatch may fail. • Caddy should forward Host and Authorization: ◦ header_up Host {host} ◦ header_up Authorization {>Authorization} Quick debugging steps to run now (paste results) 1. Tail livekit server logs, then join the room with the token and paste any dispatch/job lines you see. 2. Tail agent logs at the same time and paste the relevant lines (registered worker + any job/connection logs). 3. From the agent host, run: ◦ openssl s_client -connect rtc.v23.rh0.ds.kinetic360.io:443 -servername rtc.v23.rh0.ds.kinetic360.io ◦ and paste whether TLS handshake succeeds. If you paste the small log snippets (server dispatch lines + agent job/connection lines), I’ll read them and point to the exact fix — e.g., auth/cert mismatch, host mismatch, timing/timeout, or worker capacity. You’re very close now.
e
livekit-1 | 2026-02-25T184454.279Z INFO livekit service/roommanager.go:413 starting RTC session {"room": "kinetic", "roomID": "RM_6XB2zToQSYaY", "participant": "jlambright@app.v23.rh0.ds.kinetic360.io", "pID": "PA_hu7u4AEcNWdd", "remote": false, "room": "kinetic", "nodeID": "ND_e7YNTsosn2Dy", "numParticipants": 7, "participantInit": {"Identity": "jlambright@app.v23.rh0.ds.kinetic360.io", "Reconnect": false, "ReconnectReason": "RR_UNKNOWN", "AutoSubscribe": false, "AutoSubscribeDataTrack": "not-set", "Client": {"sdk": "FLUTTER", "version": "2.5.3", "protocol": 12, "os": "linux", "deviceModel": "Other", "browser": "Edge", "browserVersion": "145.0.0", "network": "wifi"}, "Grants": {"Identity": "jlambright@app.v23.rh0.ds.kinetic360.io", "Kind": "", "KindDetails": [], "Video": {"RoomCreate": false, "RoomList": false, "RoomRecord": false, "RoomAdmin": false, "RoomJoin": true, "Room": "kinetic", "CanPublish": true, "CanSubscribe": true, "CanPublishData": true, "CanPublishSources": [], "CanUpdateOwnMetadata": "not-set", "IngressAdmin": false, "Hidden": false, "Recorder": false, "Agent": true, "CanSubscribeMetrics": "not-set", "DestinationRoom": ""}, "SIP": {}, "Agent": {}, "Inference": {}, "Observability": {}, "RoomConfig": {"agents": [{"agentName": "ds_agent_sauron", "metadata": "<redacted>"}]}, "RoomPreset": ""}, "Region": "", "AdaptiveStream": true, "ID": "", "SubscriberAllowPause": "not-set", "DisableICELite": false, "CreateRoom": {"name": "kinetic", "agents": [{"agentName": "ds_agent_sauron", "metadata": "<redacted>"}]}, "AddTrackRequests": [], "PublisherOffer": {}, "SyncState": {}, "UseSinglePeerConnection": false}}
t
Thanks — that server log is excellent: it shows the participant join carrying the correct RoomConfiguration with an agents entry for "ds_agent_sauron". That means LiveKit did receive the dispatch instruction and attempted to create a job for an agent. Now we need to confirm the agent worker actually accepted the job and connected to the room. What this log tells us (positive) • Participant joined room "kinetic". • The participant's claims include RoomConfig.agents with agentName "ds_agent_sauron". LiveKit has the information it needs to dispatch an agent. What to check next (short checklist, high priority) 1. Agent worker registration ◦ Confirm the agent process log shows it registered with LiveKit as "ds_agent_sauron": ▪︎ Look for a line like: registered worker {"agent_name":"ds_agent_sauron", "id":"AW_..."} ◦ If you don’t see it, start the agent and watch for registration messages. 1. Agent job assignment / acceptance logs ◦ Tail the agent worker logs while a participant joins and paste lines around the join time. We want to see: ▪︎ job received or job created ▪︎ job accepted ▪︎ any errors like job rejected, auth error, or timeouts (e.g., “room connection was not established within 10 seconds”) ◦ If the agent never logs a job, LiveKit either didn’t find an eligible worker or the worker rejected the job. 1. LiveKit server dispatch/job logs ◦ Tail LiveKit server logs for dispatch-related lines when the participant joins. Look for things like: ▪︎ dispatch created / job created for agentName ds_agent_sauron ▪︎ job assigned to worker id AW_... ▪︎ job rejected or no available workers ◦ Paste those lines if you see any; they’ll identify whether LiveKit matched a worker and whether the worker accepted. 1. Confirm exact agent naming & mode ◦ If your agent entrypoint uses 
@server.rtc_session(agent_name="ds_agent_sauron")
, you are doing explicit dispatch — token matches that correctly. ◦ If the agent uses a different name, or agent_name is omitted, that mismatch prevents acceptance. Verify names match exactly (including underscores/casing). 1. Connectivity and auth checks for the agent process ◦ From the agent host, ensure it can reach the LiveKit WSS endpoint configured in its env: ▪︎ Validate env: LIVEKIT_URL (should be wss://rtc.v23.rh0.ds.kinetic360.io or ws://localhost:7880 for local), LIVEKIT_API_KEY, LIVEKIT_API_SECRET. ▪︎ Quick TLS check: openssl s_client -connect rtc.v23.rh0.ds.kinetic360.io:443 -servername rtc.v23.rh0.ds.kinetic360.io ◦ If the agent logs show auth failures, confirm API key/secret values and that Caddy or any proxy is forwarding Authorization/Host correctly. 1. Common failure signals to look for in agent logs ◦ Authentication/permission errors ◦ WebSocket connection errors ◦ Job rejected due to capacity or incompatible agent version ◦ Room connection timeouts (e.g., not connecting as a participant within expected time) What I need from you to move this to a concrete fix • Paste the agent worker log lines spanning: ◦ The time the worker registered (startup) ◦ The time a participant joined the room (when the server log above was emitted) ◦ Any lines about job received/assigned/accepted/rejected or connection timeouts • Paste any LiveKit server dispatch/job lines that mention assigning a job to a worker ID (if present). If you prefer quick actions to run now 1. Tail the agent logs, then have a user join the room (using the token you already validated). 2. Run on the server host: ◦ grep/tail for "dispatch", "job", "assigned", "registered worker", "job rejected" in the livekit logs. 1. From the agent host, run: ◦ openssl s_client -connect rtc.v23.rh0.ds.kinetic360.io:443 -servername rtc.v23.rh0.ds.kinetic360.io ◦ and paste whether the TLS handshake succeeds. I’ll read the agent logs you paste and tell you exactly whether the agent was never offered the job, rejected it, or failed to connect — and give the precise config/log fixes for that case. You’re almost there.
e
root@rtc.v23:~/livekit # redis-cli monitor | grep -i "sauron" 1772045275.100849 [0 [:1]39876] "publish" "Signal|RelaySignal|ND_e7YNTsosn2Dy|STR" "\n#type.googleapis.com/internal.Stream\x12\x9d\x05\n\x10STR_m3vMSTulResK\x12\x10REQ_dcgHN5LHBfmc\x18\xe5\xc6\xac\xe2\x9f\xb9\xe4\xcb\x18 \xe5\x82\xee\xf8\xaa\xb9\xe4\xcb\x18:\xe2\x04\x12\xdf\x04\n\xdc\x04\n\akinetic\x12'jlambright@app.v23.rh0.ds.kinetic360.io\x1a\x0fCO_aHaSjJuJTqQ5Z3\b\x04\x12\x052.5.3\x18\x0c\"\x05linux2\x05Other\x04EdgeB\a145.0.0J\x03:1R\x04wifij'jlambright@app.v23.rh0.ds.kinetic360.ior\xde\x02{\"identity\":\"jlambright@app.v23.rh0.ds.kinetic360.io\",\"name\":\"jlambright@app.v23.rh0.ds.kinetic360.io\",\"video\":{\"roomJoin\":true,\"room\":\"kinetic\",\"canPublish\":true,\"canSubscribe\":true,\"canPublishData\":true,\"agent\":true},\"roomConfig\":{\"agents\":[{\"agentName\":\"ds_agent_sauron\",\"metadata\":\"{\\\"username\\\": \\\"jlambright@app.v23.rh0.ds.kinetic360.io\\\"}\"}]}}x\x01\xa2\x01U\n\akineticrJ\n\x0fds_agent_sauron\x127{\"username\": \"jlambright@app.v23.rh0.ds.kinetic360.io\"}" 1772045399.292350 [0 [:1]39892] "subscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM" 1772046258.830265 [0 [:1]39876] "publish" "AgentInternal|CLI_5cQ3qwqrMPjM|RES" "\n%type.googleapis.com/internal.Response\x12E\n\x10REQ_V3Rj6YcIMrIP\x12\x10SRV_NLCODgx4Z5kn\x18\x95\xf5\x8a\xb9\xf0\xd5\xe4\xcb\x18:\x15\b\x01\x1a\x00\"\x0fds_agent_sauron" 1772047568.816303 [0 [:1]39892] "unsubscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM" 1772047583.627836 [0 [:1]39892] "subscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" 1772047915.685883 [0 [:1]39892] "unsubscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM" 1772047926.126999 [0 [:1]39892] "subscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" 1772048045.171709 [0 [:1]39892] "unsubscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM"
t
Thanks — this is excellent diagnostic data. The Redis traffic and your earlier token/room logs show the pieces are in place: the room join included a room_config with the agent dispatch, and the agent name 
ds_agent_sauron
is registered and listening on the correct JobRequest channels. That means LiveKit is trying to dispatch — the remaining problem is on the worker side. What the Redis output tells us (short) • The participant join published a Signal with 
roomConfig.agents
that includes 
ds_agent_sauron
— LiveKit created the dispatch job. • The agent is subscribed to the JobRequest channels for 
ds_agent_sauron
— the worker registered successfully. • The repeated unsubscribe/subscribe pattern suggests the agent process is reconnecting frequently (restarts or transient disconnects) or is not staying online long enough to accept the job. What I want you to check next (high priority — quick steps) 1) Tail the agent worker logs around a join event • Start a clean log tail on the machine running the agent, then trigger a participant join using the token. • Copy the lines from ~30s before the join to ~60s after it. We’re looking for: ◦ worker startup / "registered worker" messages ◦ "job received" / "job accepted" / "job rejected" messages ◦ any errors/exceptions, stack traces, or exit/crash logs ◦ connection timeout messages like “room connection was not established within 10 seconds” 2) Confirm the agent process is long‑running • On the agent host, run: ◦ ps or systemctl/docker inspect to confirm the agent process stays alive (not restarting). ◦ If it’s a container, check container restart count: 
docker ps --format '{{.Names}} {{.Status}}'
or 
kubectl get pods
restart counts. 3) Increase agent logging to DEBUG and add a startup heartbeat • If possible, run the agent with DEBUG logs, and add a short log line on startup and just before returning/exit so we can detect unexpected exits: ◦ Example: “sauron-agent: starting”, “sauron-agent: registered worker id=...”, “sauron-agent: waiting for job”, “sauron-agent: shutting down — reason=...” 4) Reproduce once more and capture three logs: • LiveKit server log lines at join time (you already have one). • Redis monitor lines (you have these). • Agent worker logs (missing; please paste). Common causes the logs usually show • Agent process crashes due to unhandled exception at startup or when handling a job. • Agent receives job but fails to connect to the room (auth/mTLS/cert or LIVEKIT_URL mismatch) and times out. • Proxy/timeout kills the agent’s long-lived WSS connection before job assignment. • Agent code accepts job and exits immediately (lifecycle logic issue). Quick checks you can run now (1–2 minutes) • On agent host: 
journalctl -u <agent-service> -f
or 
docker logs -f <agent-container>
while someone joins the room. • Check process restarts: 
docker inspect <container> --format '{{.RestartCount}}'
or 
kubectl get pod <agent-pod> -o jsonpath='{.status.containerStatuses[0].restartCount}'
. • From agent host verify LIVEKIT_URL and creds: ◦ print env: 
echo $LIVEKIT_URL $LIVEKIT_API_KEY | sed -e 's/./*/g'
(sanitize secret) • TLS check from agent host: 
openssl s_client -connect <http://rtc.v23.rh0.ds.kinetic360.io:443|rtc.v23.rh0.ds.kinetic360.io:443> -servername <http://rtc.v23.rh0.ds.kinetic360.io|rtc.v23.rh0.ds.kinetic360.io>
If you paste the agent log slice that spans registration and the join attempt, I’ll read it and tell you exactly: • whether the agent received the JobRequest, • if it accepted and then errored connecting to the room (and why), or • whether the worker crashed/restarted before handling the job. You’re very close — Redis shows the dispatch path is wired. Share the agent logs and I’ll point to the exact fix (lifecycle, env, auth, or proxy tweak). Sources: Agents Framework | livekit/livekit_composite | DeepWiki | Agent dispatch | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation | Server lifecycle | LiveKit Documentation | Job lifecycle | LiveKit Documentation
e
# redis-cli monitor | grep -i "sauron" 1772049571.258880 [0 [:1]41874] "subscribe" "AgentInternal|JobRequest|ds_agent_sauron|room|RCLAIM" "AgentInternal|JobRequest|ds_agent_sauron|room|REQ" 1772049599.186700 [0 [:1]41862] "publish" "Signal|RelaySignal|ND_vPFmstzsMobD|STR" "\n#type.googleapis.com/internal.Stream\x12\x9d\x05\n\x10STR_kfbc5i6o5GMQ\x12\x10REQ_oEzOwJF99rQ9\x18\x9d\xc5\x94\x9f\x8c\xb7\xe5\xcb\x18 \x9d\x81\xd6\xb5\x97\xb7\xe5\xcb\x18:\xe2\x04\x12\xdf\x04\n\xdc\x04\n\akinetic\x12'jlambright@app.v23.rh0.ds.kinetic360.io\x1a\x0fCO_roMxbh5NyccfZ3\b\x04\x12\x052.5.3\x18\x0c\"\x05linux2\x05Other\x04EdgeB\a145.0.0J\x03:1R\x04wifij'jlambright@app.v23.rh0.ds.kinetic360.ior\xde\x02{\"identity\":\"jlambright@app.v23.rh0.ds.kinetic360.io\",\"name\":\"jlambright@app.v23.rh0.ds.kinetic360.io\",\"video\":{\"roomJoin\":true,\"room\":\"kinetic\",\"canPublish\":true,\"canSubscribe\":true,\"canPublishData\":true,\"agent\":true},\"roomConfig\":{\"agents\":[{\"agentName\":\"ds_agent_sauron\",\"metadata\":\"{\\\"username\\\": \\\"jlambright@app.v23.rh0.ds.kinetic360.io\\\"}\"}]}}x\x01\xa2\x01U\n\akineticrJ\n\x0fds_agent_sauron\x127{\"username\": \"jlambright@app.v23.rh0.ds.kinetic360.io\"}"
t
Thanks — the latest Redis monitor output confirms LiveKit is publishing the job request for 
ds_agent_sauron
when a participant joins. That means the dispatch instruction is flowing end‑to‑end up to the point of job delivery. The remaining gap is on the worker: it’s either not accepting the job, reconnecting too often, or failing to connect into the room after accepting. Here’s exactly what to check and what to capture next so I can pinpoint the failure and give a targeted fix. 1) Capture the agent worker logs around a join • Tail the agent log, then have a participant join the room (use the token you generated). • Collect logs from ~30s before the join to ~60s after. Look for: ◦ Worker startup / "registered worker" lines ◦ "waiting for job" or subscribing to JobRequest ◦ "job received" / "job accepted" / "job rejected" ◦ Room connection lines (connected / failed to connect / timeouts) ◦ Any exceptions or stack traces • Paste that slice here (sanitize secrets). If you can’t paste full logs, include any lines containing "job", "register", "connect", "error", or "timeout". 2) Verify the agent process stays running (no frequent restarts) • On the agent host, check process/container restart counts: ◦ docker: 
docker ps -a
and 
docker inspect <container> --format '{{.RestartCount}}'
◦ systemd: 
systemctl status <agent-service>
◦ k8s: 
kubectl get pods
and 
kubectl describe pod <agent-pod>
• If restart count > 0 or pod/container is cycling, that explains unsubscribe/subscribe behavior you saw. 3) Confirm the agent environment and connection settings • On the agent host, confirm these env values: ◦ LIVEKIT_URL (should be your WSS endpoint, e.g., 
<wss://rtc.v23.rh0.ds.kinetic360.io>
or 
<ws://localhost:7880>
for local) ◦ LIVEKIT_API_KEY and LIVEKIT_API_SECRET (must match server) • From the agent host, run: ◦ TLS check (if using wss): 
openssl s_client -connect <http://rtc.v23.rh0.ds.kinetic360.io:443|rtc.v23.rh0.ds.kinetic360.io:443> -servername <http://rtc.v23.rh0.ds.kinetic360.io|rtc.v23.rh0.ds.kinetic360.io>
◦ Confirm WSS reachability; paste success/failure. 4) If the agent receives a job but fails to join the room, capture the connection error • Common messages to look for: ◦ “room connection was not established within 10 seconds” ◦ auth or permission errors when attempting to join ◦ WebSocket/TLS handshake failures • Those lines tell us whether the agent couldn’t authenticate, couldn’t resolve the host, or timed out. 5) Quick test you can run now • Run a minimal agent locally (if feasible) that only logs lifecycle events: ◦ on startup: print “registered worker” ◦ on job: print “job received: <room> <metadata>” ◦ on connect success/failure: print outcome • This isolates whether your production agent logic is causing failures. 6) If you want a rapid checklist to try immediately • Increase agent log level to DEBUG and restart the agent. • Reproduce a join and paste the agent log slice. • If agent restarts, share the container/pod restart reason or the crash stack trace. You’re one short step away — Redis proves dispatch is happening. Paste the agent log slice from a reproduction and I’ll tell you exactly whether the worker never saw the job, rejected it, or accepted and then failed to connect (and how to fix that).
e
How can we get the agent log?
t
Great question — I’ll walk you through how to find and capture the agent logs based on the common ways agents are run. Pick the section that matches how you deployed the agent (local process, Docker, systemd, or Kubernetes) and run the commands. If you’re not sure how it’s running, try the quick checks at the top to identify the process. Quick check — identify how the agent is running • On the agent host, run: ◦ ps aux | grep -i agent ◦ docker ps --format '{{.Names}} {{.Image}} {{.Status}}' (if you use Docker) ◦ kubectl get pods -A (if you use Kubernetes) • If you see a matching process/container/pod name, use the corresponding steps below. A. If agent runs as a local process (e.g., started via python directly) 1. Find the PID: ◦ ps aux | grep -i sauron ◦ Or search for the entrypoint: ps aux | grep -i 'agent.py' or your script name 1. Tail the stdout/stderr if run in the foreground: ◦ If it’s running in a terminal, view that terminal or tmux/screen session. 1. If it was started with nohup or redirected logs, check typical files: ◦ tail -n 200 /var/log/agent.log ◦ tail -f /path/to/your/app.log 1. If not obvious, run it in the foreground for debugging: ◦ python agent.py (or the command you use) and watch logs directly. B. If agent runs in Docker 1. List containers: ◦ docker ps 1. Identify agent container name or ID, then view logs: ◦ docker logs -f <container_name_or_id> ◦ docker logs --since "5m" <container_name_or_id> (recent logs) 1. If container is restarting, check restart count and last exit code: ◦ docker inspect <container> --format '{{.RestartCount}} {{.State.ExitCode}} {{.State.Status}}' 1. To get a continuous stream while reproducing the join: ◦ docker logs -f <container_name_or_id> & then trigger a participant join C. If agent runs as a systemd service 1. Check status: ◦ systemctl status <service-name> 1. Tail logs: ◦ journalctl -u <service-name> -f ◦ journalctl -u <service-name> --since "10 minutes ago" 1. If service crashed, see the reason in journalctl output. D. If agent runs in Kubernetes 1. List pods (identify namespace if not default): ◦ kubectl get pods -n <namespace> 1. Tail logs for the agent pod: ◦ kubectl logs -f pod/<pod-name> -n <namespace> ◦ If there are multiple containers in pod: kubectl logs -f pod/<pod-name> -c <container-name> -n <namespace> 1. If pod restarts frequently, check restart count: ◦ kubectl describe pod <pod-name> -n <namespace> 1. If you need previous logs from a crashed container: ◦ kubectl logs pod/<pod-name> -n <namespace> --previous E. What to capture (exact slices that help diagnosis) • Startup / registration lines (first 30–60s after process starts) ◦ Look for lines like: “registered worker { "agent_name": "ds_agent_sauron", "id": "AW_..." }” • The window when a participant joins ◦ Tail logs, trigger a join, then capture ~30s before to ~60s after join • Any error/exception/traceback lines • Any lines mentioning “job”, “JobRequest”, “job accepted”, “job rejected”, “room connection”, “timeout”, or WebSocket/auth failures F. Helpful commands to run and paste here • For Docker: ◦ docker logs --since "2m" <container_id_or_name> | sed -n '1,200p' • For systemd: ◦ journalctl -u <service-name> --since "2 minutes ago" | sed -n '1,200p' • For Kubernetes: ◦ kubectl logs pod/<pod-name> -n <ns> --since=120s | sed -n '1,200p' • For local process (if writing to file): ◦ tail -n 200 /path/to/logfile G. If you can’t find logs • Tell me how you start your agent (command, container image, or service name). I’ll give the exact commands to locate and stream logs from that setup. • If logs are being swallowed or the process exits immediately, run the agent manually in a foreground shell for a test (python agent.py) and paste the resulting output. Once you capture the agent log slice around a reproduction, paste it here (sanitize secrets). I’ll read it and point to exactly whether the agent never received the job, rejected it, crashed, or failed to connect — and provide the precise configuration or code changes to fix it.
e
# ps aux | grep -i sauron root 116842 4.4 2.2 9744700 760212 pts/4 Sl+ 20:02 0:06 /root/.cache/pypoetry/virtualenvs/ds-agent-sauron-7HpwCCg8-py3.12/bin/python -c import sys; from importlib import import_module; sys.argv = ['/root/.cache/pypoetry/virtualenvs/ds-agent-sauron-7HpwCCg8-py3.12/bin/sauron', 'dev', '--log-level=DEBUG']; sys.exit(import_module('ds.perception.agent.sauron.service_sauron').main()) root 117148 0.0 0.0 20116 12576 pts/4 S+ 20:02 0:00 /root/.cache/pypoetry/virtualenvs/ds-agent-sauron-7HpwCCg8-py3.12/bin/python -c from multiprocessing.resource_tracker import main;main(14) root 117149 4.7 2.2 9824700 760500 pts/4 Sl+ 20:02 0:06 /root/.cache/pypoetry/virtualenvs/ds-agent-sauron-7HpwCCg8-py3.12/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=15, pipe_handle=21) --multiprocessing-fork root 117223 0.0 0.0 3536 1804 pts/3 S+ 20:04 0:00 grep --color=auto -i sauron
t
Thanks — good confirmation. Your ps output shows the agent is running as a local Python process (the sauron entrypoint) and it’s already started in 
dev
mode with 
--log-level=DEBUG
. That’s perfect for collecting logs. Here are the simplest, reliable ways to capture the agent log output around a participant join so we can see whether the worker receives/accepts the job or crashes. If you’re sitting at the same host and the process was started interactively (pts/4) • Switch to that terminal (pts/4) if you can; you’ll likely see live DEBUG output there already. • If you started it in a terminal you can’t access, restart it in a terminal you control (below) so you can capture everything. If you can re-run the agent in a foreground terminal (recommended for debugging) 1. Stop the current agent (if safe to do): ◦ Identify PID (you have 116842) and stop it: ▪︎ kill 116842 ◦ Or gracefully stop via your supervisor if you use one. 1. Start the agent in a new terminal and redirect output to a file: ◦ cd to your project and run: ▪︎ /root/.cache/pypoetry/virtualenvs/ds-agent-sauron-*/bin/sauron dev --log-level=DEBUG 2>&1 | tee /tmp/sauron.log ◦ Or run the python command you used originally: ▪︎ python -c "import importlib, sys; sys.argv=['...','dev','--log-level=DEBUG']; importlib.import_module('ds.perception.agent.sauron.service_sauron').main()" 2>&1 | tee /tmp/sauron.log 1. In a second terminal, tail the log while you cause a participant to join: ◦ tail -f /tmp/sauron.log If the agent runs under a process manager (systemd/docker) — commands to use • Docker: ◦ docker ps | grep sauron (find container id) ◦ docker logs -f <container_id> (stream logs while reproducing) ◦ docker logs --since "2m" <container_id> | sed -n '1,200p' (capture recent slice) • Systemd: ◦ journalctl -u <service-name> -f (stream) ◦ journalctl -u <service-name> --since "5 minutes ago" (capture slice) • Kubernetes: ◦ kubectl get pods -n <ns> ◦ kubectl logs -f pod/<pod> -n <ns> ◦ kubectl logs pod/<pod> -n <ns> --previous (if it crashed) What to capture (exactly) • Startup lines (first 30s after the agent starts): we want the “registered worker” line. • The window when a participant joins: start tailing, then have the participant join; capture ~30s before to ~60s after. Look for: ◦ "registered worker" / registration details ◦ subscription to JobRequest (you’ve seen these in Redis) ◦ "job received" / "job accepted" / "job rejected" ◦ room connection logs (connected/failed/timeouts) ◦ any exceptions, stack traces, or process exit messages • If the agent process exits or restarts, capture the exit/traceback and any container/pod restart reason. How to paste logs here safely • Remove any secrets/tokens (if you see full JWTs or API secrets, redact them). • You can paste the relevant 50–200 lines; I’ll focus on lines containing words: register, job, accept, reject, connect, error, timeout, exception. If you prefer, run these exact commands now on the host and paste the output: 1. Stream logs and reproduce join: ◦ /root/.cache/pypoetry/virtualenvs/ds-agent-sauron-*/bin/sauron dev --log-level=DEBUG 2>&1 | tee /tmp/sauron.log ◦ (in another shell) tail -n 200 /tmp/sauron.log 1. Or capture a short slice after reproducing: ◦ tail -n 400 /tmp/sauron.log > /tmp/sauron_slice.txt ◦ paste the contents of /tmp/sauron_slice.txt here (sanitize secrets) Once you paste the agent log slice, I’ll read it and tell you: • whether the agent got the JobRequest, • whether it accepted and then failed to connect (and why), or • whether it crashed/restarted before handling the job — and then I’ll give the exact fix.
e
It's like the job wasn't delivered/received. root@rtc.v23:~/livekit # tail -f /tmp/sauron.log poetry run sauron dev --log-level=DEBUG ROS jazzy does not appear to be installed. fb98b31fe7086ecf5f1087add16098a6f57fbad91be49a43f2fb243602838f73 200855.172 DEBUG… asyncio Using selector: EpollSelector 200855.174 DEV … livekit.agents Watching /root/.cache/pypoetry/virtualenvs/ds-agent-sauron-7HpwCCg8-py3.12/bin 200857.165 DEBUG… asyncio Using selector: EpollSelector 200857.169 INFO … livekit.agents starting worker {"version": "1.4.2", "rtc-version": "1.1.2"} INFO … livekit.agents preloading plugins {"packages": ["livekit.plugins.silero", "av"]} 200857.171 INFO … livekit.agents HTTP server listening on :45959 200857.179 INFO … livekit.agents registered worker {"agent_name": "ds_agent_sauron", "id": "AW_NLuFXtcUWoKi", "url": "wss://rtc.v23.rh0.ds.kinetic360.io", "region": "", "protocol": 16}
t
You're almost there — the Redis and LiveKit logs show the dispatch path is wired and the agent registers correctly, but the agent isn't receiving or accepting the job at runtime. That usually means either (A) the job is never actually published to the agent channel at the exact time the agent is listening, (B) the agent is reconnecting or not long‑lived enough to accept the job, or (C) explicit dispatch rules/token mismatch. Practical next steps below to confirm exactly which of those it is and fix it. Immediate checks (do these in order — fast) 1) Verify the participant actually joined using the token you inspected • Decode the JWT that the participant used (locally or paste the payload) and confirm the payload contains: ◦ video.room == "kinetic" ◦ room_config.agents includes agentName "ds_agent_sauron" • If the joining client used a different token (or different room name), no job will be created. 2) Reproduce with live log capture (one-shot reproduce) • Tail the agent log and Redis at the same time, then trigger a join: ◦ On the agent host: tail -f /tmp/sauron.log ◦ On the server host: redis-cli monitor | grep -i sauron • Have a test client join using the verified token, then capture: ◦ Agent log lines from ~30s before to ~60s after the join ◦ Redis monitor output for the same window • Paste those exact slices. We need to see whether a JobRequest publish appears and whether the agent printed a "job received" line. 3) Check agent process health and longevity • Verify the agent process does not restart repeatedly: ◦ ps aux | grep -i sauron (you already have the process) ◦ If containerized: docker ps / kubectl get pods and inspect restart counts • If the agent is restarting or exiting, fix the crash (capture traceback from its logs). 4) Manual/CLI dispatch test • Manually create a dispatch to force a job and observe the agent: ◦ lk dispatch create --agent-name ds_agent_sauron --room kinetic --metadata '{"test": true}' • If agent receives this, then token dispatch timing or token mismatch is the problem. If not, agent connectivity/liveness is the problem. 5) Ensure explicit vs auto dispatch semantics are correct • Your agent is registered with agent_name "ds_agent_sauron" (explicit dispatch mode). That requires the token to target that exact name (which your token does). • If you want auto dispatch (agent launched for rooms without explicit agent_name in token), remove agent_name from your agent RTC entrypoint. What to capture and paste here (most useful) • Agent log slice (30s before → 60s after reproduction). Look for "registered worker", "waiting for job", "job received", "job accepted", "job rejected", connection errors, exceptions. • Redis monitor slice for the same window (you already showed similar, but paste the exact window). • LiveKit server log lines for the join event (you posted earlier and it was fine). • Output of the manual dispatch command if you run it. Common fixes depending on what logs show • If Redis shows JobRequest publish but agent log shows nothing → agent not subscribed at the publish time (agent restarted or not long-lived). Make agent persistent (fix crash / prevent exit). • If agent log shows job received but then errors connecting → check LIVEKIT_URL, API key/secret, and TLS/Host settings (Caddy header passthrough). • If agent log never shows registered worker (contradiction) → agent not actually connected to that same control plane (check LIVEKIT_URL value). • If manual dispatch succeeds but token dispatch doesn't → token mismatch (wrong room name or token used by join client), or join happens before agent subscribed (race) — pre-warm or use persistent agents. Small, practical tests to run now 1. Manual dispatch, observe agent: ◦ lk dispatch create --agent-name ds_agent_sauron --room kinetic --metadata '{"test":1}' 1. Start agent (foreground) with logs to file: ◦ /root/.cache/.../sauron dev --log-level=DEBUG 2>&1 | tee /tmp/sauron.log 1. Trigger a join with the exact token you decoded and watch logs. If you paste the agent log slice after one reproduce and the matching Redis monitor output, I’ll identify: • whether the JobRequest was published and whether the agent saw it, • if the agent accepted and then failed to connect (and why), • or if the agent never saw the job because it was offline or restarted. Which of the above captures would you like to produce now? I can walk you through the exact commands for each step. Sources: Agent dispatch | LiveKit Documentation | Job lifecycle | LiveKit Documentation
e
Is there a way to set the job type in the agent session?
t
Short answer: yes — you can control the job type, but it’s not set on the AgentSession itself. Job type is configured at the agent server/entrypoint level (the rtc_session / server options), and it determines how jobs are created and scaled (per room vs per publisher). What to change and where • Python: pass 
type
(a ServerType) on the 
@server.rtc_session()
decorator:
Copy code
from livekit.agents import Server, JobContext
from livekit.agents.server import ServerType

server = Server()

@server.rtc_session(agent_name="ds_agent_sauron", type=ServerType.ROOM)
async def entrypoint(ctx: JobContext):
    # one agent instance per room
    ...
• Node: set 
serverType
(or JobType) in your ServerOptions when constructing the agent server:
Copy code
import { ServerOptions, JobType } from "@livekit/agents";

const opts = new ServerOptions({
  // ...
  serverType: JobType.JT_ROOM, // JT_ROOM or JT_PUBLISHER
});
Which types mean what • ServerType / JT_ROOM: one agent instance per room (job scoped to the room). Use when a single agent should serve the whole room. • ServerType / JT_PUBLISHER: one agent instance per publisher (job scoped to each publisher). Use when you need a separate agent per participant/publisher. Notes and guidance • This is a deployment/lifecycle setting (how LiveKit dispatches and scales agents), not a runtime property inside the AgentSession. • If you use explicit dispatch (agent_name set and tokens specify that name), the job type still controls whether LiveKit will create a room-scoped job or a publisher-scoped job for that dispatch. • Choose room vs publisher based on whether the agent’s work is per-room (shared context) or per-publisher (isolated per participant). If you want, I can: • Suggest which server type fits your use case if you describe how your agent should behave (single shared assistant vs per-user assistants). • Show a complete minimal agent entrypoint (Python or Node) with ServerType set and a small test flow you can run locally. Which language do you prefer? Sources: Server options | LiveKit Documentation | Agent session | LiveKit Documentation | Agent dispatch | LiveKit Documentation
e
from livekit.agents import AgentServer, JobContext, WorkerOptions, cli # Note: In recent versions, WorkerType replaces ServerType for clarity from livekit.agents.worker import WorkerType
t
Thanks — good catch on the import. You can set the job type using the newer WorkerType enum (which replaces ServerType terminology). This is configured at the agent server / worker level (WorkerOptions or AgentServer setup), not inside AgentSession/AgentSession logic. What it does • WorkerType controls the scope / lifecycle of dispatched jobs: ◦ WorkerType.ROOM (one job per room) — a single agent instance serves the whole room. ◦ WorkerType.PUBLISHER (one job per publisher) — an agent is created per publisher (participant) in the room. • This affects how LiveKit dispatches and scales agents for incoming work; it’s not a runtime flag inside AgentSession. Python example — minimal patterns 1) If you create a server and run via CLI helper:
Copy code
from livekit.agents import AgentServer, JobContext, cli
from livekit.agents.worker import WorkerType, WorkerOptions

server = AgentServer()

@server.rtc_session(agent_name="ds_agent_sauron")
async def entrypoint(ctx: JobContext):
    # agent logic here
    await ctx.connect(auto_subscribe=True)
    await ctx.room.wait_for_disconnect()

if __name__ == "__main__":
    opts = WorkerOptions(worker_type=WorkerType.ROOM)  # or WorkerType.PUBLISHER
    # pass opts into whatever runner your agent uses (cli.run_app or similar)
    cli.run_app(server, worker_opts=opts)
2) If you construct and run a Worker/Server directly:
Copy code
from livekit.agents import AgentServer, cli
from livekit.agents.worker import WorkerOptions, WorkerType

server = AgentServer()
# register your rtc_session handlers...

worker_opts = WorkerOptions(worker_type=WorkerType.PUBLISHER)
# start agent server programmatically with options
cli.run_app(server, worker_opts=worker_opts)
Notes on exact parameter names • In some versions the option may be named 
type
or `server_type`; in newer releases WorkerOptions uses 
worker_type
and the enum is 
WorkerType
. • If your CLI helper doesn’t accept 
worker_opts
as shown, pass the options in the pattern your agent entrypoint expects — the examples above show the intent and canonical fields. Which type to pick • Use WorkerType.ROOM when you want a single shared assistant with room-level state. • Use WorkerType.PUBLISHER when you need isolated agents per participant (for per-user state or parallel independent agents). • If agents are currently not receiving jobs, ensure your token/dispatch and worker_type align: ◦ Explicit dispatch (token names an agent_name) will create jobs as the configured worker_type dictates. ◦ If your agent sets 
agent_name
, explicit dispatch via token or API is required. Quick checklist to apply after changing WorkerType • Confirm agent registers with the server under the same agent_name. • Tail agent logs during a join to verify job received (job lifecycle messages should appear). • If you switch to PUBLISHER, expect more concurrent jobs (one per publisher); ensure capacity is sufficient. If you’d like, I can: • Show the exact invocation to run your agent with WorkerOptions in your project (paste how you run the agent, and I’ll adapt the command). • Explain how switching WorkerType will change the Redis channels / job messages to watch for in your logs. Which would help most?
e
We couldn't import ServerType
t
Thanks — that’s expected. Recent versions replaced ServerType with WorkerType, so trying to import ServerType will fail. Use WorkerType (and WorkerOptions) instead. What to change • Replace any ServerType imports/usages with WorkerType. • Set the job scope via WorkerOptions.worker_type (or the option name your runner expects). Minimal Python example (ready to run):
Copy code
from livekit.agents import AgentServer, JobContext, cli
from livekit.agents.worker import WorkerOptions, WorkerType

server = AgentServer()

@server.rtc_session(agent_name="ds_agent_sauron")
async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=True)
    await ctx.room.wait_for_disconnect()

if __name__ == "__main__":
    opts = WorkerOptions(worker_type=WorkerType.ROOM)  # or WorkerType.PUBLISHER
    cli.run_app(server, worker_opts=opts)
Notes • WorkerType.ROOM = one agent instance per room. • WorkerType.PUBLISHER = one agent instance per publisher. • If your agent is registered with an 
agent_name
, explicit dispatch (token or API) is required; WorkerType controls how LiveKit creates those jobs (room- vs publisher-scoped). • If your CLI helper uses slightly different param names, pass the WorkerOptions object where your runner expects worker config. If you paste the exact import error or how you currently start the agent, I’ll give the precise one-line fix for your code/command. Want me to adapt the example to your current agent start command?
e
It says 
WorkerType.PUBLISHER
doesn't exist.
t
Good catch — thanks for flagging that. WorkerType.PUBLISHER doesn't exist. Here's a clear summary of how the enums map and what to use depending on what you want to achieve. Short answer • Use WorkerType for the worker process options (it does not include PUBLISHER). • Use ServerType (or the 
type
parameter on 
@server.rtc_session
) when you want per-room vs per-publisher job semantics — ServerType includes PUBLISHER. What to change (concrete) 1) If you want one agent per room (room-scoped job) • Use WorkerType.ROOM for worker options (if your runner needs WorkerOptions) • Use ServerType.ROOM (or omit type if ROOM is default) in the rtc_session decorator if needed Example (Python):
Copy code
from livekit.agents import AgentServer, JobContext, cli, ServerType
from livekit.agents.worker import WorkerOptions, WorkerType

server = AgentServer()

@server.rtc_session(agent_name="ds_agent_sauron", type=ServerType.ROOM)
async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=True)
    await ctx.room.wait_for_disconnect()

if __name__ == "__main__":
    opts = WorkerOptions(worker_type=WorkerType.ROOM)
    cli.run_app(server, worker_opts=opts)
2) If you want one agent per publisher (publisher-scoped job) • Use ServerType.PUBLISHER in the decorator (this is the correct place for PUBLISHER) • Do not try to use WorkerType.PUBLISHER — it does not exist Example (Python):
Copy code
from livekit.agents import AgentServer, JobContext, cli, ServerType
from livekit.agents.worker import WorkerOptions, WorkerType

server = AgentServer()

@server.rtc_session(agent_name="ds_agent_sauron", type=ServerType.PUBLISHER)
async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=True)
    await ctx.room.wait_for_disconnect()

if __name__ == "__main__":
    opts = WorkerOptions(worker_type=WorkerType.ROOM)  # worker process type remains ROOM/AGENT as appropriate
    cli.run_app(server, worker_opts=opts)
Why this difference exists • WorkerType describes the worker process configuration (how your agent process runs) and its enum values are limited (e.g., ROOM, AGENT). • ServerType (or the 
type
parameter on rtc_session) controls job scoping (ROOM vs PUBLISHER). That’s where PUBLISHER belongs. Quick checklist to avoid errors • Don’t import or reference WorkerType.PUBLISHER — it’s not defined. • Import ServerType and use ServerType.PUBLISHER when you need per-publisher agent jobs. • Ensure the 
agent_name
you register matches dispatch targets in tokens or API. • Restart the agent after making this change and watch for “registered worker” and “job received” logs. If you want, paste the exact imports and the command you use to start the agent and I’ll provide the exact, ready-to-run file or start command adjusted to your version of the SDK. Which would you prefer — a short file you can copy/paste to replace your current entrypoint, or the single-line start command to run with the right options? Sources: OpenAI and LiveKit partner to turn Advanced Voice into an API | Server options | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki
e
Which version of the agents sdk is this for?
t
Thanks — the right next step is to confirm which Agents SDK version you have installed so I can give precise import names and the correct enum members. Please tell me which runtime you’re using (Python or Node) and paste the output of one of these commands: • Python: ◦ pip show livekit-agents ◦ or: python -c "import pkg_resources; print(pkg_resources.get_distribution('livekit-agents').version)" • Node: ◦ npm list @livekit/agents When you paste the version, I’ll reply with: • The exact imports to use (ServerType vs WorkerType) for that version, and the available enum members (ROOM, PUBLISHER, etc.). • A short, ready-to-run agent entrypoint snippet that matches your installed SDK. • If your installed version lacks the needed API, I’ll recommend the exact upgrade command and target version to install (e.g., pip install --upgrade livekit-agents==1.4.0). Which runtime is this — Python or Node? Paste the version output and I’ll tailor the snippet. Sources: Models overview | LiveKit Documentation | An open source stack for real-time multimodal AI | An open source stack for real-time multimodal AI | Agent dispatch | LiveKit Documentation
e
v1.4.2
t
Thanks — with v1.4.2 confirmed, here’s a concise, version-accurate summary and the exact guidance you need so your agent receives jobs correctly. Key facts for v1.4.2 • Use the Python package 
livekit-agents
v1.4.2. • The enum you should use to control per-room vs per-publisher job scope is set at the rtc_session/server level (ServerType or the decorator’s 
type
), not on AgentSession internals. • The WorkerType enum (used in WorkerOptions) is separate and does not include a PUBLISHER value; WorkerType is about the worker process, not job scoping. • Because your agent registers with an 
agent_name
(ds_agent_sauron), explicit dispatch is required (token or API must target that agent_name). What to do now (exact, prioritized steps) 1) Confirm ServerType import and use • For v1.4.2 use ServerType for job scoping in the decorator:
Copy code
from livekit.agents import AgentServer, JobContext, cli, ServerType
from livekit.agents.worker import WorkerOptions, WorkerType

server = AgentServer()

@server.rtc_session(agent_name="ds_agent_sauron", type=ServerType.ROOM)  # or ServerType.PUBLISHER
async def entrypoint(ctx: JobContext):
    await ctx.connect(auto_subscribe=True)
    await ctx.room.wait_for_disconnect()

if __name__ == "__main__":
    opts = WorkerOptions(worker_type=WorkerType.ROOM)
    cli.run_app(server, worker_opts=opts)
• Use ServerType.PUBLISHER when you need one agent instance per publisher (participant). Use ServerType.ROOM for one instance per room. 2) Ensure your token dispatch matches (you already have this working) • Your token must contain a RoomConfiguration with RoomAgentDispatch that specifies 
agent_name: "ds_agent_sauron"
. • You already showed the token payload includes this — good. 3) Verify agent process is long‑lived and not cycling • Your agent registered successfully (log shows registered worker). The next gap is the agent not seeing the job at runtime. • Ensure the agent process does not restart during a join. Keep it running in foreground for a test: ◦ Stop current process and run: 
sauron dev --log-level=DEBUG 2>&1 | tee /tmp/sauron.log
◦ Tail logs: 
tail -f /tmp/sauron.log
and reproduce a join. 4) Manual dispatch test (fast, isolates token vs delivery) • Run (on a host with 
lk
CLI configured to your control plane):
Copy code
lk dispatch create --agent-name ds_agent_sauron --room kinetic --metadata '{"test":true}'
• Watch the agent logs. If the agent receives this job, your worker is capable and the remaining issue is token timing or the join path. If it does not, the problem is worker connectivity/liveness. 5) Capture the exact log window for diagnosis • When reproducing, collect: ◦ agent log slice (30s before → 60s after join) ◦ livekit server log for the join ◦ redis monitor slice (you already have helpful output) • Paste the agent log slice here if the agent still doesn’t log “job received” or “job accepted”. Common pitfalls (based on your traces) • agent_name set → requires explicit dispatch (you are using explicit dispatch in tokens — correct). • Agent process restarts or disconnects between registration and job publish (your Redis subscribe/unsubscribe pattern hints at reconnects). • Token used by the joining client doesn’t match the room or doesn’t include the room_config (you validated token earlier — ensure the same token is actually used by the client that joined). If you want, I can: • Provide a minimal drop-in agent entrypoint for v1.4.2 that logs lifecycle events (registered, waiting for job, job received, job accepted, connect success/fail) so you can run it and paste the logs. • Walk you through the manual dispatch command and how to interpret the agent logs live. Which would you prefer? I can paste the small minimal agent file you can run now to capture the needed logs. Sources: Agent dispatch | LiveKit Documentation | server-sdk-kotlin | livekit/livekit_composite | DeepWiki | Job lifecycle | LiveKit Documentation | Examples | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki
e
livekit-1 | 2026-02-25T211830.613Z INFO livekit.transport remotebwe/remote_bwe.go:214 remote bwe: channel congestion detected, applying channel capacity update {"room": "kinetic", "roomID": "RM_uy6jgg5rCuAJ", "participant": "oherbert@app.v23.rh0.ds.kinetic360.io", "pID": "PA_YCxfZQVoMen9", "remote": false, "transport": "SUBSCRIBER", "reason": "LOSS", "old(bps)": 547248, "new(bps)": 1034295, "lastReceived(bps)": 2011200, "expectedUsage(bps)": 1089522, "commitThreshold(bps)": 1035045, "channel": {"name": "non-probe", "estimate": {"name": "non-probe-estimate", "startTime": "2026-02-25T211443.376Z", "elapsed": "3m47.237722186s", "numSamples": 1872, "samples": [{"value": 2011200, "sinceFirst": "0s"}], "lowestValue": 413818, "highestValue": 2011200, "kendallsTau": 0, "direction": "INCONCLUSIVE"}, "nack": {"name": "non-probe-nack", "windowStartTime": "2026-02-25T211828.484Z", "windowDuration": "2.129518332s", "packets": 363, "repeatedNacks": 46, "nackRatio": 0.12672176308539945}, "channelTrend": "CONGESTING", "channelCongestionReason": "LOSS"}}
t
Thanks — that LiveKit transport log shows a network congestion (BWE) event for a subscriber; it’s useful for call quality troubleshooting but it’s not the reason jobs aren’t being delivered to your agent. Good to have it, though — keep watching transport logs for unrelated media problems. Let’s refocus on the remaining issue: job dispatch reaches Redis but your agent process isn’t showing a “job received”/“job accepted” log. Here are the exact, minimal steps to capture the missing agent-side evidence and the most likely fixes depending on what we find. 1) Reproduce while streaming the agent logs (one-shot) • On the agent host, run the agent in foreground and save logs: ◦ Stop existing agent process (kill the PID you listed). ◦ Start in a terminal you control: /root/.cache/.../sauron dev --log-level=DEBUG 2>&1 | tee /tmp/sauron.log • In another terminal start Redis monitor (optional): ◦ redis-cli monitor | grep -i sauron • Now have a client join the room using the exact token you decoded. 2) Capture and paste the agent log slice • After reproducing, run: ◦ sed -n '1,200p' /tmp/sauron.log (or tail -n 200 /tmp/sauron.log) • Paste the lines from ~30s before join to ~60s after. • We’re specifically looking for: ◦ “registered worker” (you already have this) ◦ Subscription to job channels / “waiting for job” ◦ “job received” or “job accepted” ◦ Any exceptions, connection timeouts, or shutdown messages 3) If the agent log still shows no job received, run a manual dispatch • On a machine with the LiveKit CLI configured, run: lk dispatch create --agent-name ds_agent_sauron --room kinetic --metadata '{"test":true}' • Watch /tmp/sauron.log for immediate “job received” lines. ◦ If manual dispatch produces logs, token-based dispatch/timing is the problem (race/timing or incorrect token used by the client). ◦ If manual dispatch also produces nothing, the agent is not subscribed/connected correctly even though it registered — check for process restarts or channel subscription issues. 4) Quick process-health checks (do these now) • Confirm no restarts: ps aux | grep -i sauron (verify uptime) • If containerized: docker ps / docker inspect <container> --format '{{.RestartCount}}' • Check for early crashes in /tmp/sauron.log (tracebacks) 5) Likely outcomes and fixes • Agent never sees JobRequest but Redis shows publish: ◦ Cause: agent restarted / disconnected between registration and job publish. Fix: make agent long-lived, prevent restarts, add retry/backoff in agent startup. • Agent subscribes but rejects job immediately: ◦ Cause: job validation in agent code (metadata/room mismatch) or worker capacity rules. Fix: inspect agent acceptance logic and logs for “rejected”. • Agent accepts job but fails to join room: ◦ Cause: LIVEKIT_URL/credentials mismatch, TLS/Host issues via proxy (Caddy), or connection timeout. Fix: confirm LIVEKIT_URL (wss://rtc.v23...), API key/secret, and test openssl s_client from agent host. 6) Helpful commands to run and paste • Start agent and capture logs (one command you already ran): /root/.cache/.../sauron dev --log-level=DEBUG 2>&1 | tee /tmp/sauron.log • After reproducing: tail -n 400 /tmp/sauron.log • Manual dispatch: lk dispatch create --agent-name ds_agent_sauron --room kinetic --metadata '{"test":true}' • TLS check from agent host: openssl s_client -connect rtc.v23.rh0.ds.kinetic360.io:443 -servername rtc.v23.rh0.ds.kinetic360.io If you paste the agent log slice you capture after one reproduce (or the result of the manual dispatch attempt), I’ll read it and tell you exactly: • whether the job was published but missed due to a restart, • whether the agent received and rejected the job (and why), or • whether the agent accepted but failed to join the room (and what to change). Which capture do you want to do first — reproduce + paste the agent log slice, or try manual dispatch and paste the agent output? I’ll guide you step-by-step for either.
2 Views
Open in Slack
PreviousNext
Powered by