https://livekit.io logo
Join Slack
Powered by
If I deploy an agent, while existing sessions work...
# ask-ai
n
If I deploy an agent, while existing sessions work fine for 1h as stated with the docs, there seem to be a little gap, around ~15 secs after a successful deploy, in which new sessions will shutdown just after starting, likely starting with the previous deploy although I'm not sure
✅ 1
t
Thanks for reaching out, Francisco. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m glad you reported this. You mentioned new sessions briefly shutting down (~15s) right after a successful agent deploy. I’m going to help narrow down whether this is expected drain/health-check behavior or something that needs deeper investigation. First, a short summary of what I’m addressing: • You see existing sessions keep working for ~1h as expected, but new sessions started within ~15 seconds after a deploy tend to close immediately. To diagnose this I need a few key details. Please provide as many of the following as you can: 1. Are you running on LiveKit Cloud or a self‑hosted LiveKit/agent deployment? 2. Which deployment method/tooling did you use for this deploy? (Cloud dashboard, kubectl/Helm, Docker Compose, CI/CD pipeline, etc.) 3. One example timeline: approximate deploy time and the timestamp when a new session started and immediately shut down. 4. Server/agent logs around that deploy time (especially any SIGTERM/SIGINT, readiness probe results, or health-check responses). Client/SDK logs or disconnect error messages for the affected session are also very helpful. 5. What exactly happens when the session “shuts down”? Any SDK error codes/messages or disconnect reasons you see on the client? 6. How many agent/server instances were running during the deploy (single instance vs. rolling update across many instances)? 7. Do you have custom values for 
drain_timeout
, 
terminationGracePeriodSeconds
, or similar termination settings? If self‑hosted, please share those config values. 8. Do you see any health-check failures (HTTP 503) or readiness probe failures around the time of deploy? 9. Which SDK and server/agent versions are you using? 10. Can you reproduce the issue reliably? If yes, what are the exact steps to reproduce? Why I’m asking these: • When an agent is updated, servers enter a draining state that stops new job assignments while allowing active jobs to complete. Drain timeouts and readiness/health probes determine when traffic is routed to new instances. If the new instance isn’t fully ready or routing hasn’t stabilized, new sessions can fail immediately. I need logs, timestamps, and config to tell whether this is normal draining/health-check timing or an unexpected readiness/routing gap. What I can do next once you reply: • If you provide logs/timestamps and config, I’ll look for SIGTERM/drain events, health-check responses, and routing behavior around the deploy time to determine the root cause. • If it looks like a readiness gap on new instances, I’ll suggest concrete mitigations (for example, increasing termination/rollout grace periods or ensuring readiness checks succeed before traffic is routed). • If it looks like something else (routing instability, or an agent not reporting ready), I’ll point to specific remedial steps you can apply. If you prefer, start by sharing whether you’re on LiveKit Cloud or self‑hosted and one example timestamp + the relevant client/server log snippets — that’s usually enough to get a quick diagnosis. Sources: Deploy and scale agents on LiveKit Cloud | LiveKit | Agents on LiveKit Cloud | Deploy voice AI on global infrastructure | LiveKi... | Server options | LiveKit Documentation | Server startup modes | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation, with interactive elements
n
lk cloud, dockerfile built on lk server
t
Thanks — that helps. You’re on LiveKit Cloud and using a custom Dockerfile built on the LiveKit server image. I’ll help narrow this down so we can determine whether that ~15s gap is expected drain/health-check behavior or a readiness/routing problem we can fix. I still need a few targeted pieces of info to diagnose quickly. Please share as many of these as you can: 1. A single example timeline (timestamps, approximate): when you triggered the deploy and when a new session started and immediately shut down. 2. The exact Dockerfile base image and agent/server version you used (the image tag or `git`/version string). 3. Any server/agent logs around that deploy time—especially lines showing SIGTERM/SIGINT, "draining", readiness probe results, or health-check responses. If you can’t paste full logs, a 30–60s snippet before/after the deployment event is ideal. 4. The client/SDK disconnect error or log message for the session that closed immediately (SDK error code or text). 5. How many agent/server instances were running during the deploy (single instance vs. multiple instances)? 6. Which deploy method you used on LiveKit Cloud (dashboard deploy vs. CI/CD pushing the image) and whether the deploy was a rolling update. 7. Any custom termination/timeout settings you changed (for example 
drain_timeout
in your agent config or K8s 
terminationGracePeriodSeconds
). If none, say “default”. 8. Whether this is reproducible every deploy and, if so, the reproducible steps. Why these details matter: • SIGTERM/drain and health-check entries will show whether the old instance was draining and whether the new instance marked itself ready before traffic routed. • Client disconnect messages show whether the session was rejected at connect time (routing/readiness) or disconnected after being accepted (server-side shutdown). • Knowing instance count and rollout type shows whether traffic may briefly route to an instance that’s not fully initialized. If you’d like a quick test you can run while gathering the above: • Try a deploy during a low-traffic window and tail the agent logs plus the client log during the 30s window around deploy. Look for the new instance returning a 200 on 
/
(healthy) before it receives the first session. If the new instance is not returning 200 when it receives traffic, that points to a readiness timing issue. Next steps after you share the info: • I’ll check logs for drain health checks and readiness timing and tell you whether this is expected behavior. If it looks like a readiness/routing gap, I’ll recommend concrete mitigations (for example increasing rollout/termination grace periods or ensuring the health endpoint returns 200 only after initialization completes). Send the example timestamp + the relevant log snippets and client error text and I’ll analyze them. , with interactive elements
3 Views
Open in Slack
PreviousNext
Powered by