i am on a paid plan and my livekit agent deployed ...
# ask-ai
k
i am on a paid plan and my livekit agent deployed on livekit cloud is answering calls successfully, but on the UI it says pending. lk agent status says updating
t
Thanks for reaching out, Ethan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding about your agent showing "pending" in the UI while
lk agent status
says "updating" even though calls are answered. This is expected during a rolling deployment: the new agent version can serve traffic while old instances are still draining, so the UI/CLI can show "pending" or "updating" until all old instances finish. That’s why calls work but the status hasn’t settled. Please try these quick checks: • Confirm routing: are new sessions going to the new instances (look for recent session start logs tied to the new version)? • Check agent logs for drain activity (search for "draining worker" or SIGTERM handling). • Verify old-instance session counts drop to zero (per-version active sessions). • Note timing: cloud rollout + default
drain_timeout
can take up to ~30–60 minutes to fully finish. If you’d like, tell me: • the agent version(s) shown in the UI/CLI, • any recent deployment time, • and any drain-related log lines you see— and I’ll help interpret the logs and next steps. Sources: Deployment reliability on LiveKit Cloud | LiveKit, with interactive elements
k
why does it seem to be oscillating between "running" and "updating" though?
t
Thanks — good question. Oscillation between "running" and "updating" usually means the control plane is seeing instances start, stop, or change state repeatedly during a rollout or restart cycle. Common causes: • Rolling deployment behavior: new instances come up while old ones drain; if draining and start-up overlap, status can flip between states. • Long drain or stuck shutdown: if old instances take too long to exit (blocked SIGTERM handling or long-running sessions), the system will repeatedly mark the agent as still updating. • Failing health checks or intermittent starts: instances that briefly fail health checks or restart will cause repeated transitions. • Frequent deployments or autoscaling churn: many rapid deploys or scale events can keep toggling the state. What I can help diagnose next — could you share: • the agent version(s) listed and the recent deployment time, • a sample of agent logs around a transition (look for "draining worker", SIGTERM, or health-check failures), • whether you’ve seen many quick restarts or autoscale events? With those details I’ll point to the most likely root cause and steps to stabilize the status. , with interactive elements
k
tthis happened even on first deployment
t
Thanks — I see you’re asking why the status oscillates on first deployment. That’s frustrating; I’ll help narrow it down. On a first deployment this flip-flop commonly happens because the control plane and the new agent instances are still synchronizing. Likely causes: • Health-check timing/startup latency: the agent process may not pass health checks right away, so the control plane marks it as updating then running as checks pass/fail. • Container/process lifecycle signals: if SIGTERM/SIGINT aren’t handled or the process takes time to initialize, the system may repeatedly register state changes. • Rapid instance churn during rollout: instances briefly start, fail, restart, or are re-provisioned, causing repeated transitions. • Drain behavior overlapping startup: even on first deploy, internal rollout logic can trigger brief draining/handovers that show as updating. To diagnose quickly, please share: • the agent version shown in the UI/CLI, • the deployment timestamp (when you first deployed), • a short excerpt of agent logs around one transition (search for lines with "draining", "SIGTERM", or health-check failures). With those I’ll point to the most likely root cause and recommend concrete fixes to stabilize the status. , with interactive elements
k
lk agent status is bounching between running and updating
t
Thanks — I’m responding about the
lk agent status
bouncing between "running" and "updating." That oscillation is usually caused by startup/health-check timing, signal handling, or brief instance churn during rollout (even on first deploy). To pinpoint the root cause I need a few details: • Agent version shown in the UI/CLI • Deployment timestamp (when you first deployed) • A short log excerpt around a transition (look for lines mentioning "draining", "SIGTERM", health-check failures, or restarts) Send those and I’ll analyze the logs and recommend the exact next steps to stabilize the status. , with interactive elements
k
there's nothing of note in the build logs
t
Thanks — I hear you: builds look clean but
lk agent status
still bounces between "running" and "updating." That typically means the control plane is seeing the instance start, fail a health check, or briefly restart — so we need runtime/health-check details (build logs won't show these). Could you provide the following so I can diagnose quickly? • Agent version shown in the UI/CLI • Deployment timestamp (when you first deployed) • A short excerpt of the agent runtime logs around a transition (look for lines mentioning health checks,
draining
,
SIGTERM
, restarts, or errors) • Confirm whether the health-check endpoint responds with HTTP 200 and how long it takes to respond If you paste the log excerpt here I’ll analyze it and recommend the concrete next steps to stop the oscillation. , with interactive elements
k
agent id: CA_TiQxgTh9eux4 first deploy timestamp: May 5, 2026, 112654 AM not sure how to get the others
t
Thanks — I’m looking at your agent ID (CA_TiQxgTh9eux4) and first deploy time. Since you don’t have runtime logs or the agent version yet, I’ll walk you through the minimal checks that let me diagnose the oscillation. Please do the following and paste the results here: 1. Get the agent version and current status output ◦ Run:
lk agent status CA_TiQxgTh9eux4
◦ Paste the full output. 1. Tail the agent runtime logs around a transition ◦ If you have CLI access where the agent runs, run: ▪︎
lk agent logs CA_TiQxgTh9eux4 --follow --lines 200
◦ If your environment uses
kubectl
, run: ▪︎
kubectl logs -l app=livekit-agent -c agent --since=10m --tail=200
◦ Paste any lines that mention
drain
,
SIGTERM
,
health
,
error
, or restart messages (even short snippets are helpful). 1. Check the health-check endpoint ◦ If you know the agent’s health URL, run: ▪︎
curl -i <HEALTH_URL>
◦ If unknown, tell me whether your agent is behind a load balancer or the cloud provides a health-check path and I’ll guide you to find it. 1. Note timing of a recent transition ◦ When you next see the status flip, capture the exact timestamp and include logs from ~1 minute before to 2 minutes after. If you prefer, paste the outputs you can get now (even just the
lk agent status
output) and I’ll start analyzing. , with interactive elements
k
the agent is hosted on livekit cloud, what is the health url i don't see anything in the logs of use
t
Thanks — I’ll help you find the health URL and collect the runtime info we need to diagnose the bouncing status. First: on LiveKit Cloud the agent health endpoint is available from the agent details page in the Cloud Console and is typically one of
/health
or
/healthz
. Please do the following: 1. Open the LiveKit Cloud Console and navigate to your agent (ID: CA_TiQxgTh9eux4). On the agent details page look for a Health or Endpoints section — the full health URL will be shown there (it will look like
https://.../health
or
https://.../healthz
). 2. Once you have the health URL, run: ◦ curl -i HEALTH_URL Paste the full response here (status code and body). A healthy agent should return HTTP 200 quickly. 1. Grab runtime logs from the cloud Console: ◦ On the same agent details page there should be a Logs tab or a “View logs” link. Open it and filter for the last 30 minutes. ◦ Copy any lines around a status transition (search for
drain
,
draining
,
SIGTERM
,
health
,
error
, or
restart
). If you can’t find those terms, paste the last ~200 log lines. 1. If you can’t find the health URL or logs in the Console, tell me what you see on the agent details page (tabs/links available) and I’ll guide you step-by-step. When you paste the health-check response and any log excerpts, I’ll analyze them and tell you the next concrete steps to stop the oscillation. , with interactive elements
k
{"message": "registered worker", "level": "INFO", "name": "livekit.agents", "agent_name": "inbound-agent-theinsurancesaver-b", "id": "CAW_yuKAKMycg8bS", "url": "", "region": "US East B", "protocol": 17, "timestamp": "2026-05-05T181347.028946+00:00"} {"message": "draining worker", "level": "INFO", "name": "livekit.agents", "id": "CAW_yuKAKMycg8bS", "timeout": 1800, "timestamp": "2026-05-05T181818.498898+00:00"} {"message": "shutting down worker", "level": "INFO", "name": "livekit.agents", "id": "CAW_yuKAKMycg8bS", "timestamp": "2026-05-05T181818.499552+00:00"}
t
Thanks — I reviewed the logs you pasted. They show an instance (ID CAW_yuKAKMycg8bS) registering, then entering drain and shutdown shortly after: • "registered worker" at 2026-05-05T181347 • "draining worker" and "shutting down worker" at 2026-05-05T181818 This explains the oscillation: the control plane is performing a rollout where a new instance comes up while the old one drains. The UI/CLI will flip between "running" and "updating" until the old instance finishes draining (the runtime
drain_timeout
is 1800s by default). Because calls are still answered, the new instance is serving traffic correctly. Next steps I recommend: • Confirm a new instance is active (check the agent list for a different worker ID than CAW_yuKAKMycg8bS). • Check the agent health endpoint (from the agent details page in the Cloud Console) returns HTTP 200 quickly. • Monitor logs for any repeated drain/shutdown cycles or health-check failures that would cause continuous churn. If you want, paste the agent status output (
lk agent status CA_TiQxgTh9eux4
) and the most recent 200 log lines from the Cloud Console and I’ll check for repeated cycles or health-check errors and suggest specific fixes. , with interactive elements
k
the issue is my last deployment was ~160000
t
Thanks — I’m responding about your deployment at ~16:00 causing the oscillation. Your logs show a worker registered at 181347 then immediately entered drain/shutdown at 181818. That pattern (new worker coming up while an old worker drains) explains the status flipping: the control plane reports "updating" while instances are draining and "running" when a healthy instance is active. The default
drain_timeout
is 1800s (30 minutes), so the UI/CLI can continue toggling until the draining instance fully exits or active sessions drop to zero. Next, please check these two quick things and paste the results so I can confirm the cause and advise fixes: 1. Current worker list and statuses ◦ Run:
lk agent status CA_TiQxgTh9eux4
◦ Paste the full output so I can see all worker IDs and their states. 1. Recent logs to detect repeated churn ◦ From the Cloud Console logs for this agent, paste the last ~200 lines (or any repeated blocks showing
registered
,
draining
,
shutting down
, health failures, or restarts). If the agent list shows a new worker ID active and no repeated drain cycles, the status should stabilize after the remaining drain timeout. If you see repeated register→drain cycles in the logs or health-check failures, paste those lines and I’ll recommend targeted fixes (health-check tuning, investigate SIGTERM handling, or increasing stability during startup). , with interactive elements
Thanks — I reviewed the info you shared. What’s happening • Your logs show a worker (CAW_yuKAKMycg8bS) registering at 182854 and then immediately entering drain/shutdown. That indicates a rollout/drain cycle is in progress. • The UI/CLI flip between "running" and "updating" because the control plane sees new instances come up while old instances are draining. The default drain timeout is 1800s (30 minutes), so status can oscillate until draining instances finish. What to check next (quick) 1. Confirm a different active worker ID (new worker ≠ CAW_yuKAKMycg8bS): ◦ Paste the full output of
lk agent status CA_TiQxgTh9eux4
. 1. Confirm health endpoint responds 200: ◦ From the agent details page in Cloud Console, get the health URL (likely
/health
or
/healthz
) and run: ▪︎ curl -i HEALTH_URL ◦ Paste the response. 1. Watch logs for repeated register→drain cycles: ◦ Paste the last ~200 log lines from the Cloud Console (look for repeated
registered
,
draining
,
shutting down
, health failures). If the agent list shows a new worker active and you only see one drain cycle, the status should stabilize after the remaining drain timeout. If you see repeated register→drain cycles or health-check failures, paste those log lines and I’ll recommend targeted fixes (health-check tuning, investigate SIGTERM handling, or startup health latency). , with interactive elements
k
I don't see any other worker ids
t
Thanks — I see you don’t have any other worker IDs. That means the control plane saw a worker register and then immediately drain/shutdown instead of a healthy replacement coming up, which causes the status to flip. Most likely causes: • The new instance fails a health check or crashes soon after startup. • Startup latency causes health checks to fail before readiness is reported. • SIGTERM/shutdown logic is triggered (or cloud rollout immediately marked it draining). Please do these two quick checks and paste the results here: 1. Full logs after the last "registered worker" line (from that timestamp forward — include any shutdown/error lines). 2. Health endpoint response: ◦ Find the health URL on the agent details page (likely
/health
or
/healthz
) and run: ▪︎ curl -i HEALTH_URL ◦ Paste the response (status code and body). If you prefer, paste the last ~200 log lines from the Cloud Console and I’ll scan them for the exact failure reason and recommend the specific fix (health-check tuning, restart, or redeploy). , with interactive elements
k
how long do health checks poll for?
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!