Thanks — that’s a clear description and helpful context. You mentioned seeing a “participant disconnected” event, then an immediate call to

ctx.shutdown()

, and in <1% of cases the process hangs for 10–15 minutes before being killed. Below are the most likely scenarios that cause

ctx.shutdown()

not to complete promptly, and practical steps to investigate and mitigate each one. What can cause

ctx.shutdown()

to hang • Long-running shutdown callbacks ◦ Any callback registered with

ctx.add_shutdown_callback()

runs during shutdown. If a callback does slow work (large DB writes, blocking I/O, synchronous network calls, heavy serialization, etc.), shutdown will wait for that work to finish and can appear to hang. • Shutdown timeout too short or too long depending on workload ◦ The agent waits up to the configured

shutdown_process_timeout

for callbacks to complete before killing the process. If your callbacks usually finish but occasionally block (e.g., transient DB latency), you’ll see rare long shutdowns or forced kills. • Blocking vs. non-blocking shutdown usage ◦

ctx.shutdown()

is non-blocking and returns immediately; if code relies on awaiting a close operation but doesn’t use an awaitable close (e.g.,

aclose()

), subsequent in-process work may prolong process exit or leave tasks running. • Drain behavior delaying exit ◦ If shutdown is invoked with

drain=True

, the agent first waits for queued/in-flight work (like speech processing or queued jobs) to finish. Long or stuck tasks in the queue will delay shutdown. • External dependencies stalling during shutdown ◦ Calls to external services (webhooks, APIs, databases) from shutdown hooks or finalizers that experience timeouts or network issues can hang shutdown until those operations time out. • Rare edge cases in cleanup code ◦ Deadlocks, resource contention, or unexpected exceptions not handled inside shutdown callbacks can prevent callback termination and delay shutdown. Practical investigation steps 1. Identify what your shutdown callbacks are doing ◦ List all callbacks registered with

ctx.add_shutdown_callback()

and note what external calls or blocking operations they perform. 1. Add timing/logging around shutdown hooks ◦ Log start/finish timestamps for each shutdown callback so you can pinpoint which callback or action is slow in the rare cases. 1. Capture stack traces when hanging ◦ When you detect a hang, capture a thread/async stack trace of the process (SIGQUIT or equivalent) to see where it’s blocked. 1. Check for external latency patterns ◦ Correlate hangs with upstream service metrics (DB, webhook endpoints, network) to see if external slowness coincides with the delayed shutdowns. 1. Verify whether

drain=True

is used ◦ If you use draining, inspect queued work for long-running tasks and consider bounds/limits on queue size or processing time. 1. Review shutdown timeout configuration ◦ Check your

shutdown_process_timeout

setting. If your normal cleanup sometimes legitimately takes longer than the default window, increase it; otherwise, keep it short and make callbacks faster. 1. Ensure shutdown hooks are non-blocking or bounded ◦ Make hooks small and quick. For longer cleanup, consider: ▪︎ Offloading to a background worker or a durable queue (so shutdown remains quick). ▪︎ Using non-blocking async calls with explicit timeouts. ▪︎ Wrapping expensive operations with a short timeout and retrying outside shutdown. Immediate mitigation suggestions • Make shutdown callbacks lightweight: save minimal state synchronously; enqueue heavier work for background processing. • Add per-call timeouts for any network or DB work inside shutdown hooks. • Increase

shutdown_process_timeout

only if you expect legitimate longer cleanup and can tolerate longer graceful shutdowns. • If using

drain=True

, consider not draining on participant disconnects unless necessary, or set tighter limits on queued task durations. If you’d like, I can help with next steps: • Review your registered shutdown callbacks (paste the list or snippets) and suggest which to shorten or move off-process. • Propose logging instrumentation you can add to pinpoint the slow callback. • Suggest configuration values for

shutdown_process_timeout

based on typical callback durations you share. Which of those would you like to try first, and can you share what your shutdown callbacks currently do? Sources: Job lifecycle | LiveKit Documentation