{"message": "process exiting", "level": "INFO", "name": "livekit.agents", "reason": "", "pid": 44567, "job_id": "AJ_eEQ4C8FKLRaH", "room_id": "RM_qn5xv7GHFCFw", "timestamp": "2026-03-11T124039.430618+00:00"}, can we handle some custom logic on receiving the process exiting event, is there a way to handle it?

Thanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

do we get the JobContext on ctx.add_shutdown_callback(my_cleanup_task)

Thanks — I’m responding to your question about whether

JobContext

is available inside a shutdown callback registered with

ctx.add_shutdown_callback()

. Yes — the

JobContext

is accessible inside shutdown callbacks. When you define a callback in the scope of

entrypoint(ctx: JobContext)

, the callback closes over

ctx

and can use

ctx

(for example

ctx.job

,

ctx.room

, etc.) to perform cleanup, logging, or other shutdown tasks. Example:

async def entrypoint(ctx: JobContext):
    async def my_shutdown_hook():
        <http://logger.info|logger.info>(f"Cleaning up job {ctx.job.id} in room {ctx.room.name}")
        # use ctx to save state, call APIs, close resources, etc.
        await some_api_client.aclose()

    ctx.add_shutdown_callback(my_shutdown_hook)

Practical notes: • Callbacks are awaited during shutdown and must finish within the

shutdown_process_timeout

(default 10 seconds). Increase that timeout if your tasks need more time. • For long-running work, consider enqueueing work to an external service from the shutdown hook so the agent can exit promptly. • You can also use

finally

blocks or explicit close calls for API clients to ensure resources are cleaned up. If you want, tell me what specific data or resources you need to access from

ctx

during shutdown (e.g., room ID, participant list, job metadata, API clients) and I’ll provide a tailored snippet and timeout recommendations. Sources: Job lifecycle | LiveKit Documentation | Build a multilingual voice agent that automatically switches languages | Li...

provide a tailored snippet and recommendations for timeouts and error handling in python

Thanks — I’ll provide a concise, tailored Python snippet showing how to register shutdown callbacks with

JobContext

, plus practical recommendations for timeouts and error handling. Example snippet (Python)

import asyncio
import logging
from livekit.agents import JobContext  # adjust import path as needed

logger = logging.getLogger(__name__)

async def entrypoint(ctx: JobContext):
    # Example of an API client created per job
    api_client = create_api_client()  # replace with your client constructor

    # Graceful close helper with timeout and error handling
    async def safe_close_client():
        try:
            # If your client has an async close method:
            await asyncio.wait_for(api_client.aclose(), timeout=3.0)
            <http://logger.info|logger.info>("API client closed successfully")
        except asyncio.TimeoutError:
            logger.warning("Timeout while closing API client")
        except Exception as e:
            logger.exception("Error while closing API client: %s", e)

    # Cleanup task that uses ctx for job/room info and performs final work
    async def my_shutdown_hook():
        try:
            <http://logger.info|logger.info>("Shutdown hook starting for job=%s room=%s", ctx.job.id, getattr(ctx.room, "name", None))

            # Example: gather state, send a webhook, and close API client.
            # Wrap any external call with timeout.
            summary = collect_local_usage_summary()
            try:
                await asyncio.wait_for(send_webhook(summary), timeout=4.0)
                <http://logger.info|logger.info>("Webhook sent during shutdown")
            except asyncio.TimeoutError:
                logger.warning("Webhook timed out during shutdown; enqueued for retry")
                enqueue_retry_job(summary)
            except Exception as e:
                logger.exception("Webhook failed during shutdown: %s", e)
                enqueue_retry_job(summary)

            # Ensure API client is closed (safe)
            await safe_close_client()

        except Exception as e:
            # Catch-all to prevent an unhandled exception from aborting other callbacks
            logger.exception("Unhandled error in shutdown hook: %s", e)

    # Register the shutdown callback
    ctx.add_shutdown_callback(my_shutdown_hook)

    # Normal entrypoint work continues...
    try:
        await run_job_loop(ctx)
    finally:
        # Additional per-job finally cleanup if desired
        await safe_close_client()

Recommendations — timeouts • Keep shutdown callbacks short and bounded. Default

shutdown_process_timeout

is 10 seconds; design callbacks to finish well within that window. • Use

asyncio.wait_for(..., timeout=X)

around external I/O (webhooks, DB writes, client closes). Example timeouts: ◦ Close lightweight API clients: 2–5 seconds ◦ Webhook or HTTP call: 3–6 seconds ◦ Aggregation/local state writes: 1–2 seconds • If you need more time for shutdown tasks, increase

shutdown_process_timeout

in your agent server options, but prefer enqueuing longer jobs to external workers. Recommendations — error handling and reliability • Wrap each logical step in its own try/except so a failure in one step doesn’t stop other cleanup actions. • Treat shutdown hooks as a best-effort opportunity: if an external call fails or times out, record the necessary state locally (or enqueue a retry) rather than blocking shutdown indefinitely. • Use idempotent operations when possible; retries should be safe. • Log at appropriate levels (info/warn/error) and include job and room identifiers to aid debugging. • Prefer enqueueing follow-up work (e.g., a background queue, task broker, or durable job) from the shutdown hook for anything that may take longer than the shutdown window. When to increase shutdown timeout vs. offload work • Increase

shutdown_process_timeout

only for short, essential tasks that truly need to finish before exit (final DB flush, releasing a lock). • Offload lengthy processing (reports, large uploads) by enqueuing a job or calling an external service from the shutdown hook; this keeps agent shutdown responsive and reliable. If you want, I can: • Provide a variant of the snippet that uses a specific HTTP client (e.g.,

aiohttp

) or a concrete retry/enqueue implementation. • Suggest exact configuration lines for increasing

shutdown_process_timeout

in your deployment if you tell me how you run agents (Kubernetes, systemd, etc.).

it run on kubernetes

Thanks — I’ll respond to your request for a Kubernetes-specific Python snippet and recommendations for timeouts and error handling when using shutdown callbacks in LiveKit Agents. Below is a tailored example plus deployment guidance and best practices. 1) Tailored Python snippet (Kubernetes-ready)

import asyncio
import logging
from livekit.agents import JobContext

logger = logging.getLogger(__name__)

# Example entrypoint called per job
async def entrypoint(ctx: JobContext):
    api_client = create_api_client()  # replace with your client
    usage_state = {}

    async def safe_close_client():
        try:
            await asyncio.wait_for(api_client.aclose(), timeout=3.0)
            <http://logger.info|logger.info>("API client closed")
        except asyncio.TimeoutError:
            logger.warning("Timeout while closing API client")
        except Exception:
            logger.exception("Error closing API client")

    async def my_shutdown_hook():
        try:
            <http://logger.info|logger.info>(
                "Shutdown hook start job=%s room=%s",
                getattr(ctx.job, "id", None),
                getattr(ctx.room, "name", None),
            )

            # Example: collect local state (quick)
            try:
                summary = collect_local_usage_summary(usage_state)
            except Exception:
                logger.exception("Failed to collect usage summary")
                summary = None

            # Example: send webhook with bounded timeout
            if summary is not None:
                try:
                    await asyncio.wait_for(send_webhook(summary), timeout=4.0)
                    <http://logger.info|logger.info>("Shutdown webhook sent")
                except asyncio.TimeoutError:
                    logger.warning("Webhook timed out; enqueuing retry")
                    enqueue_retry_job(summary)
                except Exception:
                    logger.exception("Webhook error; enqueuing retry")
                    enqueue_retry_job(summary)

            # Ensure API client is closed (safe)
            await safe_close_client()

        except Exception:
            # Prevent exceptions from aborting other callbacks
            logger.exception("Unhandled error in shutdown hook")

    # Register shutdown callback (ctx is available inside the callback)
    ctx.add_shutdown_callback(my_shutdown_hook)

    # Normal job work
    try:
        await run_job_loop(ctx, usage_state)
    finally:
        # Best-effort cleanup if entrypoint exits normally
        await safe_close_client()

2) Kubernetes-specific deployment recommendations • terminationGracePeriodSeconds must exceed your drain timeout and any expected shutdown work: ◦ Example: if

drain_timeout

= 2400s (40 minutes), set

terminationGracePeriodSeconds

to something slightly larger (e.g., 2500). • Configure the agent server with an appropriate drain_timeout so the agent finishes active sessions before termination:

from livekit.agents import AgentServer
server = AgentServer(drain_timeout=2400)

• Ensure Pod disruption or rollout procedures allow enough time for long-lived sessions when you expect long drains. 3) Timeout strategy (practical defaults) • API client close: 2–5 seconds • Outbound webhook/HTTP call: 3–6 seconds • Local aggregation / small DB writes: 1–2 seconds • Keep the full shutdown path comfortably under your configured

shutdown_process_timeout

(default 10s unless increased). If you rely on the Kubernetes drain window, ensure both

drain_timeout

and Pod

terminationGracePeriodSeconds

are set accordingly. 4) Error handling & reliability patterns • Wrap each logical step in its own try/except so one failure doesn’t stop other cleanup steps. • Treat shutdown hooks as "best-effort": on external failure/timeout, persist state locally and enqueue a retry job or push to a durable queue. • Make operations idempotent where possible so retries are safe. • Log job and room identifiers in every log line to make post-mortem easier. • If shutdown tasks may exceed the allowed window, offload long work to an external worker (enqueue a job) from the shutdown hook rather than blocking agent exit. 5) When to increase timeouts vs offloading work • Increase

shutdown_process_timeout

or

drain_timeout

only for short, essential tasks that must finish before exit (final DB flush, release locks). • Offload long processing (reports, large uploads) by enqueuing to a durable worker from the shutdown callback. If you want, I can: • Provide a concrete example using

aiohttp

for the webhook call and a simple retry/enqueue implementation. • Suggest exact YAML edits for

terminationGracePeriodSeconds

and a Kubernetes readiness/liveness change to coordinate draining behavior. Which of those would help most next? Sources: Python Agents: Handling exceptions in your entrypoint | LiveKit | Deployment reliability on LiveKit Cloud | LiveKit | Job lifecycle | LiveKit Documentation

will ctx.shutdown handler called, if there is an exception?

You’re asking whether a shutdown callback registered with

ctx.add_shutdown_callback()

will still run if an exception occurs during the session. Short answer: yes. Why and what to expect • Shutdown callbacks are executed as part of the agent’s teardown phase regardless of how the session ends — whether normal shutdown,

ctx.shutdown()

call, or because an exception caused the job to terminate. They’re intended to provide a reliable place for cleanup (save state, send webhooks, close clients, etc.). • Callbacks are awaited by the framework during shutdown, so they will run even after an error, but they must finish within the shutdown timeout window (see notes below). Practical recommendations • Wrap each step in the shutdown callback with its own try/except so one failure won’t prevent other cleanup tasks from running. • Use bounded timeouts (e.g.,

asyncio.wait_for(...)

) around external I/O (webhooks, DB writes, closing clients) to avoid blocking shutdown. • Persist minimal state locally or enqueue a retry job if an external call fails, rather than retrying indefinitely inside the shutdown hook. • Log job/room identifiers and errors so you can trace failures during shutdown. Example pattern

async def entrypoint(ctx: JobContext):
    async def shutdown_hook():
        try:
            # quick local work
            summary = collect_local_usage()
        except Exception:
            logger.exception("collect_local_usage failed")

        try:
            await asyncio.wait_for(send_webhook(summary), timeout=4.0)
        except asyncio.TimeoutError:
            logger.warning("Webhook timed out; enqueuing retry")
            enqueue_retry(summary)
        except Exception:
            logger.exception("Webhook failed; enqueuing retry")
            enqueue_retry(summary)

        try:
            await asyncio.wait_for(api_client.aclose(), timeout=3.0)
        except Exception:
            logger.exception("Error closing api_client")

    ctx.add_shutdown_callback(shutdown_hook)
    # ... main job work ...

If you want, I can: • Provide a concrete

aiohttp

webhook example with retry/enqueue logic. • Suggest exact timeout values and how to align them with your Kubernetes

terminationGracePeriodSeconds

and the agent

drain_timeout

. Which would you prefer? Sources: Job lifecycle | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki

if the process shutdown time expires will the ctx.shutdown get called?

You’re asking whether a shutdown callback registered with

ctx.add_shutdown_callback()

will still run (or finish) if the process shutdown window expires. Short answer: the callbacks are started when `ctx.shutdown()`/process teardown begins, but they must complete within the configured shutdown window — if the window expires the process will be forcefully terminated and incomplete callbacks will be interrupted. Details and what to expect • When shutdown begins (e.g.,

ctx.shutdown()

called, SIGTERM from Kubernetes, or room/session end), the framework runs all registered shutdown callbacks. • The framework waits only for the configured shutdown timeout (default: 10 seconds). If callbacks do not finish before that timeout, the process is terminated and any unfinished callbacks are stopped. • In Kubernetes, the Pod’s

terminationGracePeriodSeconds

must also cover the agent’s drain/shutdown time. If the Pod grace period ends, the kubelet sends SIGKILL and no further code will run. Practical recommendations • Keep each shutdown callback short and bounded. Use

asyncio.wait_for(..., timeout=...)

around external I/O (HTTP/webhook calls, DB writes, client close) so you don’t block the shutdown window. • Wrap each step in its own try/except so one failure doesn’t prevent other cleanup logic from running. • Persist minimal state locally or enqueue a retry/durable job if an external call can’t complete within the window, rather than attempting long retries inside the shutdown hook. • If you truly need more time for essential work, increase the agent’s

shutdown_process_timeout

(and align Kubernetes

terminationGracePeriodSeconds

and any

drain_timeout

you use) so the process has enough time to finish cleanup. • Prefer offloading lengthy work to an external worker by enqueuing a job from the shutdown callback — this keeps shutdown responsive and predictable. If you want, I can: • Provide a short example showing

asyncio.wait_for

usage in your shutdown hook (with

aiohttp

webhook example), and sample values to use for

shutdown_process_timeout

and

terminationGracePeriodSeconds

based on your expected jobs. Which would you prefer? Sources: Job lifecycle | LiveKit Documentation

{"message": "exiting forcefully", "level": "WARNING", "name": "livekit.agents", "timestamp": "2026-03-13T061426.866682+00:00"} in this case how to handle the call gracefully?

Thanks — I’m responding to your log showing "exiting forcefully" and your question about handling that case gracefully. What “exiting forcefully” means • That log indicates the process reached the end of its allowed shutdown window and was terminated (for example the agent’s shutdown callbacks didn’t finish before the shutdown timeout or Kubernetes pod grace period expired). • When that happens, any in-flight cleanup may be interrupted and not complete. What you can do to make shutdowns graceful and resilient • Design shutdown hooks as short, bounded, and idempotent so they reliably run within the allowed window. • Use timeouts around each external operation so a slow third-party call doesn’t block everything. • Persist minimal, essential state locally and enqueue long work to an external worker instead of doing it synchronously in shutdown hooks. • Align LiveKit timeouts and Kubernetes pod grace periods so the OS gives your process enough time to drain. Concrete steps to implement 1) Use bounded operations in shutdown hooks • Wrap every external call with

asyncio.wait_for(..., timeout=X)

. • Example timeouts: close client 2–5s, webhook 3–6s, short DB/write 1–2s. 2) Break up cleanup so one failure won’t block other steps • Use separate try/except blocks for each cleanup action. • Log failures and enqueue retries to a durable queue or DB when an external call fails. 3) Persist essential state locally before external calls • Flush minimal state synchronously (fast disk write) so you can resume or retry later even if the process is killed. • Then attempt non-essential network calls; if they fail or time out, push the task to a worker for retry. 4) Offload long-running tasks • From the shutdown hook, enqueue a job (e.g., to Redis, SQS, Kafka) that a separate worker will process. That way shutdown stays fast and reliable. 5) Align timeouts with Kubernetes • Set LiveKit agent options: ◦

shutdown_process_timeout

— how long the process waits for shutdown callbacks (default 10s). Increase only if absolutely needed. ◦

drain_timeout

— how long the agent will continue sessions before draining. • Set Kubernetes Pod spec: ◦

terminationGracePeriodSeconds

>

drain_timeout

and greater than the total of the expected shutdown window. • Example: if `drain_timeout = 2400`s (40 min), set

terminationGracePeriodSeconds

to ~2500s so Kubernetes won’t SIGKILL before your drain completes. 6) Detect and react to forced-exit in logs/monitoring • Emit structured logs and metrics when shutdown callbacks start/finish and on errors. • Alert when you see "exiting forcefully" so you can adjust timeouts or investigate slow external dependencies. Minimal Python pattern (summary) • Collect local minimal state • Persist locally (fast write) • Enqueue long work • Attempt bounded network calls with per-call timeouts • Close clients with bounded timeouts • Wrap each step in try/except and log errors If you’d like, I can: • Provide a concrete Python snippet using

aiohttp

for a webhook with

asyncio.wait_for

, plus an example of enqueueing to Redis for retries. • Suggest specific values for

shutdown_process_timeout

,

drain_timeout

, and

terminationGracePeriodSeconds

based on your typical session lengths. Which would you prefer: a code example (aiohttp + Redis enqueue) or recommended timeout values tailored to your workload?