Is there a way to handle graceful rolling deployments by ensuring that when a rolling deployment is happening, the livekit worker pods acknowledge the SIGTERM and not accept any new jobs?

Thanks for reaching out, Bharath. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

so I run the workers on kubernetes and code is in python

Thanks — responding to your Kubernetes + Python setup. Below are concrete steps, a recommended Deployment YAML snippet, and best practices to ensure Python LiveKit worker pods receive SIGTERM, mark themselves draining, and stop accepting new jobs. Key points to implement • Start the agent in production/graceful mode so it handles SIGTERM (example:

uv run agent.py start

). • Ensure the agent process is PID 1 in the container (no signal-swallowing wrappers) or use

exec

in your entrypoint so SIGTERM reaches it. • Configure

drain_timeout

to the maximum time you want pods to wait for active jobs to finish (defaults to 1800s). Recommended Kubernetes Deployment snippet (adjust image, names, envs)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: livekit-agent
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        app: livekit-agent
    spec:
      terminationGracePeriodSeconds: 2000   # >= drain_timeout + buffer
      containers:
      - name: agent
        image: your-registry/livekit-agent:latest
        args: ["agent.py", "start", "--drain-timeout", "1800"]
        # Or if using uv: args: ["run", "agent.py", "start", "--drain-timeout", "1800"]
        env:
        - name: ENVIRONMENT
          value: "production"
        # ensure liveness/readiness probes reflect acceptance of new jobs
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 30

Important configuration and behavior • terminationGracePeriodSeconds must be greater than or equal to your

drain_timeout

plus a buffer so Kubernetes waits long enough for the agent to finish draining. • When the pod receives SIGTERM, the agent sets itself to draining and stops accepting new jobs (status WS_FULL), and waits up to

drain_timeout

for running jobs to complete. • Use readiness probes that reflect worker readiness. When draining starts the agent will mark itself unavailable so Kubernetes (and load balancers) stop routing new work to it. • If you wrap the agent in a shell script, ensure the script uses

exec

to start the agent (so SIGTERM is forwarded), e.g.,

exec uv run agent.py start --drain-timeout 1800

. Verification • Watch logs for the "draining worker" message when you scale down or deploy. • Test by triggering a rolling update and verifying new pods accept new sessions while old pods log drain messages and finish existing jobs. If you want, I can: • Provide a tailored Deployment manifest with your image name and specific

drain_timeout

value, or • Suggest readinessProbe endpoints/commands based on your container’s health endpoints. Which would you prefer?, with interactive elements

my agent isn't wrapped in any shell script. Also i start my agent like this: AgentSession().start() method

Thanks — responding to your Python + Kubernetes setup where you start the agent with

AgentSession().start()

and have no shell wrapper. Good news: that setup is compatible with LiveKit’s graceful shutdown. Actions to ensure rolling deployments drain correctly: 1. Ensure the agent process receives SIGTERM ◦ You already have no wrapper, so the Python process will get SIGTERM. If you ever add an entrypoint script, start the agent with

exec

so signals propagate. 1. Run the agent in production/graceful mode / enable drain timeout ◦ Start the runtime that wires up shutdown handling (production start mode). Configure

drain_timeout

to the max time pods should wait for jobs to finish (default 1800s). If you’re invoking programmatically, ensure the AgentSession/AgentServer is initialized with the drain timeout option or start via the production CLI that accepts

--drain-timeout

. 1. Kubernetes pod settings ◦ Set terminationGracePeriodSeconds >=

drain_timeout

+ buffer. ◦ Use RollingUpdate with conservative `maxUnavailable`/`maxSurge` values. ◦ Make readinessProbe reflect worker readiness so pods are removed from rotation when draining. 1. Verify behavior ◦ Start active jobs, send SIGTERM to the agent process (or trigger a rolling update/scale-down), and confirm logs show “draining worker,” the worker stops accepting new jobs (status becomes WS_FULL), and running jobs finish or time out by

drain_timeout

. If you’d like, share: • how you instantiate AgentSession (code snippet), and • the

drain_timeout

you prefer and your current Pod spec (terminationGracePeriodSeconds), and I’ll give a minimal code example showing where to set the drain timeout and a tailored Kubernetes Deployment snippet. Sources: Deployment reliability on LiveKit Cloud | LiveKit | Deployment reliability on LiveKit Cloud | LiveKit | Server startup modes | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation, with interactive elements

If my drain_timeout is defaulted to 1800s and my terminationGracePeriodSeconds is 300 is that a problem? can you explain how effectivley I configure this and why?

Short answer: yes — that’s a problem. terminationGracePeriodSeconds must be at least as long as the agent’s drain_timeout (plus a small buffer) for graceful draining to complete. Why: • When Kubernetes wants to stop a pod it sends SIGTERM and then waits terminationGracePeriodSeconds before force-killing the container with SIGKILL. • The LiveKit agent, on SIGTERM, enters drain mode and waits up to

drain_timeout

for in-flight and running jobs to finish. • If terminationGracePeriodSeconds (300s) is shorter than drain_timeout (1800s), Kubernetes will SIGKILL the pod before the agent’s drain period finishes, aborting running jobs and preventing a clean handoff. How to configure correctly: • Option A — Increase Kubernetes grace period: ◦ Set terminationGracePeriodSeconds >= drain_timeout + buffer (e.g., drain_timeout 1800s → terminationGracePeriodSeconds 1900–2000s). • Option B — Decrease agent drain_timeout: ◦ Set

--drain-timeout

(or the AgentServer/AgentSession parameter) to a value <= terminationGracePeriodSeconds minus buffer (e.g., if you want 300s pod grace, set drain_timeout ≤ 270–290s). • Add a small buffer (10–20s) so the process can perform final cleanup before SIGKILL. Practical examples: • If you keep drain_timeout default 1800s: ◦ Pod spec: terminationGracePeriodSeconds: 2000 • If you want pods to terminate within ~5 minutes: ◦ Start agent with:

--drain-timeout 270

(or set the equivalent in AgentSession/AgentServer) ◦ Pod spec: terminationGracePeriodSeconds: 300 Verification steps: 1. Update values. 2. Trigger a rollout/scale-down with active jobs. 3. Confirm logs show “draining worker” and jobs finish before container exit. 4. Ensure no SIGKILL in logs (would show abrupt termination). If you tell me which you prefer (increase pod grace or reduce drain_timeout) and your desired shutdown window, I’ll provide the exact command or Deployment snippet to apply. , with interactive elements

what exactly happens with

drain_timeout

? Even if a call is in progress after the

drain_timeout

also, will the call be terminated abruptly?

Great question — here’s exactly what

drain_timeout

does and what happens to in-progress calls. What

drain_timeout

does • When the agent receives SIGTERM it enters drain mode: it marks the worker unavailable for new jobs (

WS_FULL

), finishes any in-flight job launches, and then waits for running jobs to complete. •

drain_timeout

is the maximum time the agent will wait for those running jobs to finish before it proceeds with shutdown. What happens when

drain_timeout

expires • If running jobs (calls/sessions) have not completed by the end of

drain_timeout

, the agent will stop waiting and the process will continue shutdown. If the Kubernetes pod’s terminationGracePeriodSeconds has already elapsed, Kubernetes will send SIGKILL and the container will be forcefully terminated. • In practice that means any still-active calls can be terminated abruptly when the agent gives up waiting and the container is killed. Practical implications • To avoid abrupt termination of in-progress calls, ensure: ◦ terminationGracePeriodSeconds >=

drain_timeout

+ small buffer (10–30s) so Kubernetes does not SIGKILL the container before the agent finishes draining. ◦

drain_timeout

is set to a value that matches how long you’re willing to wait for calls to finish (e.g., average/maximum call length). • If you need a strict upper bound on pod shutdown time, reduce

drain_timeout

to that bound (and set pod grace a little larger). If you want to prioritize letting calls finish, increase pod grace to match a larger

drain_timeout

. If you want, tell me your typical call length and desired maximum shutdown window and I’ll recommend concrete

drain_timeout

and terminationGracePeriodSeconds values. , with interactive elements

why do I need both

drain_timeout

and

terminationGracePeriodSeconds

? terminationGracePeriodSeconds also hold the pod for those amout of seconds and not accept new jobs right?

Good question — they both affect shutdown timing but play different roles, and you need both to guarantee graceful drains. • What each controls ◦

drain_timeout

(agent): How long the LiveKit agent will wait, after receiving SIGTERM, for in-flight launches to finish and for running jobs/calls to complete before it stops waiting and proceeds with shutdown. ◦

terminationGracePeriodSeconds

(Kubernetes): How long kubelet waits between sending SIGTERM and force-killing the container with SIGKILL. • Why you need both ◦ Kubernetes only enforces the time window for the container process (SIGTERM → SIGKILL). It does not know about LiveKit’s internal drain logic or job state. The agent must be allowed enough time to perform its drain procedure; Kubernetes must be configured to wait at least that long. If Kubernetes’ grace period is shorter than the agent’s

drain_timeout

, Kubernetes will SIGKILL the pod before the agent finishes draining, causing abrupt termination. ◦ Conversely, if you set a long pod grace period but a very short

drain_timeout

, the agent will stop waiting early and proceed with shutdown even though Kubernetes would have waited longer — that can still abruptly cut calls. • How they work together (correct relationship) ◦ terminationGracePeriodSeconds >= drain_timeout + small buffer (10–30s). This lets the agent finish its drain work and do cleanup before Kubernetes enforces SIGKILL. • What actually happens to in-progress calls ◦ While the agent is within

drain_timeout

, it will keep running existing calls/sessions. After

drain_timeout

elapses the agent stops waiting; if the process then exits or Kubernetes reaches its grace window, remaining calls may be terminated abruptly (SIGKILL will forcibly stop the container). • Practical guidance ◦ Pick the behavior you want: ▪︎ Let long calls finish: set

drain_timeout

to expected max call duration and set terminationGracePeriodSeconds slightly larger. ▪︎ Bound shutdown time: pick a shorter

drain_timeout

≤ desired shutdown window and set terminationGracePeriodSeconds slightly larger. ◦ Always add a small buffer (10–30s) to terminationGracePeriodSeconds beyond the

drain_timeout

. ◦ Verify by testing: start calls, trigger SIGTERM/rolling update, watch for “draining worker” logs and that no SIGKILL occurs before jobs finish. Would you like a recommended pair of values based on your typical call length and desired shutdown window? , with interactive elements

Kubernetes will send a SIGTERM and then terminationGracePeriodSeconds starts right? After this the drain_timetout starts correct? After the pod received this SIGTERM will new jobs be accpeted by this pod or not?

Short answer: Kubernetes sends SIGTERM first; the LiveKit agent begins draining immediately on SIGTERM (stops accepting new jobs), and the agent then waits up to

drain_timeout

for running jobs to finish while Kubernetes’ countdown runs from

terminationGracePeriodSeconds

. They run in parallel — Kubernetes does not wait for

drain_timeout

to start. Details: • Sequence: a. Kubelet sends SIGTERM to the container (this is immediate). b. The agent receives SIGTERM and enters drain mode right away: it sets itself unavailable (status becomes WS_FULL) and will not accept new jobs. c. Kubernetes starts the

terminationGracePeriodSeconds

timer. When that timer expires, kubelet sends SIGKILL. d. Meanwhile the agent waits up to

drain_timeout

for in-flight launches and running jobs to complete. If

drain_timeout

elapses before jobs finish, the agent stops waiting and proceeds with shutdown; if Kubernetes’ grace period elapses first, the container is force-killed. • Key point:

terminationGracePeriodSeconds

and

drain_timeout

run concurrently and both must be configured so the agent has enough time to finish draining. Ensure: ◦ terminationGracePeriodSeconds >= drain_timeout + small buffer (10–30s), or ◦ lower drain_timeout to fit within your desired pod shutdown window. If you want, tell me your desired shutdown window or typical call length and I’ll recommend concrete

drain_timeout

and

terminationGracePeriodSeconds

values. , with interactive elements

can I set drain_timeout using any env variable?

Short answer: No —

drain_timeout

cannot be set via an environment variable. How to set it: • Programmatically when creating the server:

server = AgentServer(drain_timeout=2400)  # seconds
AgentSession(server=server).start()

• Or via CLI in production start mode:

uv run agent.py start --drain-timeout 2400

Notes: • Default is 1800s (30 minutes). • Keep Kubernetes'

terminationGracePeriodSeconds

>=

drain_timeout

+ 10–30s buffer so the pod isn’t SIGKILL'd before draining completes. If you want, share how you instantiate AgentSession in your code and your desired shutdown window and I’ll show the exact change to make. Sources: Deployment reliability on LiveKit Cloud | LiveKit | Deployment reliability on LiveKit Cloud | LiveKit | Server options | LiveKit Documentation | Server startup modes | LiveKit Documentation, with interactive elements