{"log":" 130814.521 INFO … livekit.agents closing agent session due to \n","stream":"stdout","time":"2026-04-20T130814.52967095Z"} {"log":" participant disconnect (disable via \n","stream":"stdout","time":"2026-04-20T130814.52970474Z"} {"log":" `RoomInputOptions.close_on_disconn… \n","stream":"stdout","time":"2026-04-20T130814.52971048Z"} {"log":" {\"room\": \"intew-1342115558\", \n","stream":"stdout","time":"2026-04-20T130814.532561187Z"} {"log":"\"participant\": \"+mobile_number\", \"reason\": \"CLIENT_INITIATED\", \"pid\": 580, \n","stream":"stdout","time":"2026-04-20T130814.532592177Z"} {"log":"\"job_id\": \"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130814.532633428Z"} {"log":" 130814.526 DEBUG… livekit.agents stream closed \n","stream":"stdout","time":"2026-04-20T130814.534246949Z"} {"log":" {\"participant\": \"+mobile_number\", \n","stream":"stdout","time":"2026-04-20T130814.534275969Z"} {"log":"\"source\": \"SOURCE_MICROPHONE\", \"pid\": 580, \"job_id\": \"AJ_MKx6Erbcfjxn\", \n","stream":"stdout","time":"2026-04-20T130814.534281599Z"} {"log":"\"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130814.534287049Z"} {"log":" 130815.633 INFO … interview-caller Call call-4077025474 ended \n","stream":"stdout","time":"2026-04-20T130815.641585483Z"} {"log":" successfully \n","stream":"stdout","time":"2026-04-20T130815.641625014Z"} {"log":" {\"pid\": 580, \"job_id\": \n","stream":"stdout","time":"2026-04-20T130815.642034719Z"} {"log":"\"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130815.642052459Z"} {"log":" 130815.634 DEBUG… livekit.agents tools execution completed \n","stream":"stdout","time":"2026-04-20T130815.646260403Z"} {"log":" {\"speech_id\": \"speech_7b1591c2a718\", \n","stream":"stdout","time":"2026-04-20T130815.646362695Z"} {"log":"\"pid\": 580, \"job_id\": \"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130815.646371265Z"} {"log":" 130815.636 DEBUG… livekit.agents input stream detached \n","stream":"stdout","time":"2026-04-20T130815.649067339Z"} {"log":" {\"participant\": \"+mobile_number\", \n","stream":"stdout","time":"2026-04-20T130815.649805739Z"} {"log":"\"source\": \"SOURCE_UNKNOWN\", \"accepted_sources\": [\"SOURCE_MICROPHONE\"], \"pid\": \n","stream":"stdout","time":"2026-04-20T130815.649823219Z"} {"log":"580, \"job_id\": \"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130815.649870539Z"} {"log":" 130817.642 DEBUG… livekit.agents session closed \n","stream":"stdout","time":"2026-04-20T130817.646338656Z"} {"log":" {\"reason\": \"participant_disconnected\", \n","stream":"stdout","time":"2026-04-20T130817.646371966Z"} {"log":"\"error\": null, \"pid\": 580, \"job_id\": \"AJ_MKx6Erbcfjxn\", \"room_id\": \n","stream":"stdout","time":"2026-04-20T130817.646378946Z"} {"log":"\"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130817.646384786Z"} {"log":" 130817.667 DEBUG… livekit.agents uploading session report to LiveKit \n","stream":"stdout","time":"2026-04-20T130817.673675916Z"} {"log":" Cloud \n","stream":"stdout","time":"2026-04-20T130817.673746877Z"} {"log":" {\"pid\": 580, \"job_id\": \n","stream":"stdout","time":"2026-04-20T130817.674242693Z"} {"log":"\"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130817.674259843Z"} {"log":" 130824.785 ERROR… livekit.agents process did not exit in time, \n","stream":"stdout","time":"2026-04-20T130824.789291408Z"} {"log":" killing process \n","stream":"stdout","time":"2026-04-20T130824.789351788Z"} {"log":" {\"pid\": 580, \"job_id\": \n","stream":"stdout","time":"2026-04-20T130824.789729003Z"} {"log":"\"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130824.789741203Z"} {"log":" 130824.789 INFO … livekit.agents killing process \n","stream":"stdout","time":"2026-04-20T130824.792366127Z"} {"log":" {\"pid\": 580, \"job_id\": \n","stream":"stdout","time":"2026-04-20T130824.792739592Z"} {"log":"\"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130824.792752722Z"} {"log":" 130824.792 INFO … livekit.agents sending SIGUSR1 signal to process \n","stream":"stdout","time":"2026-04-20T130824.795021091Z"} {"log":" {\"pid\": 580, \"job_id\": \n","stream":"stdout","time":"2026-04-20T130824.795409866Z"} {"log":"\"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130824.795558138Z"} {"log":" 130824.865 ERROR… livekit.agents process exited with non-zero exit \n","stream":"stdout","time":"2026-04-20T130824.869527736Z"} {"log":" code -10 \n","stream":"stdout","time":"2026-04-20T130824.869560186Z"} {"log":" {\"pid\": 580, \"job_id\": \n","stream":"stdout","time":"2026-04-20T130824.86989242Z"} {"log":"\"AJ_MKx6Erbcfjxn\", \"room_id\": \"RM_yCVN7TpF2f56\"}\n","stream":"stdout","time":"2026-04-20T130824.86990636Z"} {"log":" 201209.924 WARNI… livekit.agents failed to connect to livekit, \n","stream":"stdout","time":"2026-04-20T201209.928531908Z"} {"log":" retrying in 0s \n","stream":"stdout","time":"2026-04-20T201209.928574119Z"} these are my logs from livekit agent my process is completed but the process has some issue and did not send a callback after ending the call

async def save_transcript_and_notify():
            try:
                raw_conversation = {}
                try:
                    if "session" in dir() or "session" in vars():
                        raw_conversation = session.history.to_dict() if hasattr(session, "history") else {}
                except NameError:
                    raw_conversation = {}
                
                processed_transcript = process_transcript(raw_conversation, call_id, phone_number, ctx.room.name, request_id)
                if isinstance(processed_transcript.get("conversation"), list):
                    clean_transcript = processed_transcript
                else:
                    clean_transcript = create_clean_transcript(processed_transcript)

                if isinstance(processed_transcript.get("conversation"), list):
                    clean_transcript = processed_transcript
                else:
                    clean_transcript = create_clean_transcript(processed_transcript)
                
                # Disabled to save on egress costs
                recording_file_path = ""

                transcript_data = {
                    "request_id": request_id,
                    "call_id": call_id,
                    "phone_number": phone_number,
                    "company_ctx": company_ctx,
                    "num_questions":num_questions,
                    "room_name": ctx.room.name,
                    "start_time": start_time,
                    "end_time": datetime.datetime.now(),
                    "raw_conversation": raw_conversation,
                    "processed_transcript": clean_transcript,
                    "recording_url": recording_file_path,
                    "metadata": {
                        "prompt": prompt,
                        "first_msg": first_msg,
                        "last_msg": last_msg,
                        "prompt_type": prompt_type,
                        "supplementary_questions": supplementary_questions,
                        "trunk_id": trunk_id
                    },
                    "processed_at": datetime.datetime.now()
                }
                
                await db.save_call_transcript(transcript_data)
                <http://logger.info|logger.info>(f"Transcript saved to MongoDB for call {call_id}")
                
                await db.update_call_status(call_id, "completed", {
                    "completed_at": datetime.datetime.now(),
                    "transcript_saved": True
                })
                
                if callback_url:
                    await send_callback_notification(call_id, callback_url, transcript_data)
                    
            except Exception as e:
                logger.error(f"Error saving transcript: {e}")

above is my function for transcription processing.. the audio is there in s3 but its has not sended callback

Thanks for reaching out, Blue. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Give me code snippets for how i can Increase shutdown timeout so that my whole process can be completed? and also for other options that required code

Below are concise Python and Node.js code snippets you can drop into your agent to (1) increase shutdown timeouts so your transcript + callback can finish, (2) register a shutdown hook that reliably awaits saving/transmitting the transcript, and (3) add safe HTTP callback logic with timeout+retries. I’ll keep each example minimal — pick the parts that match your setup. 1) Increase drain timeout and shutdown hook timeout (Python)

# agent_server.py
from livekit.agents import AgentServer
import asyncio

async def finalize_transcription_and_send_callback(call_id, callback_url, transcript_data):
    # use your real logic here (see HTTP example below)
    await send_callback_notification(call_id, callback_url, transcript_data)

async def entrypoint(ctx):
    async def shutdown_hook():
        # ensure the finalization runs and is awaited
        await finalize_transcription_and_send_callback(call_id, callback_url, transcript_data)

    ctx.add_shutdown_callback(shutdown_hook)
    # normal job logic here...

# Configure server: longer drain timeout and a little more time for shutdown hooks
server = AgentServer(
    drain_timeout=2400,              # wait up to 40 minutes for jobs to finish
    shutdown_process_timeout=30,     # allow shutdown hooks up to 30s
)
server.run(entrypoint)

2) Ensure your process receives signals (startup script)

#!/usr/bin/env bash
set -euo pipefail
# start the agent process with exec so signals are propagated
exec python agent_server.py

3) Graceful session shutdown usage (inside job when ending)

# when you decide to shutdown a session
await session.shutdown(drain=True)  # request draining before exit

4) Safe async HTTP callback with timeout + retries (Python, aiohttp)

import aiohttp
import asyncio

async def send_callback_notification(call_id, callback_url, payload,
                                     timeout_sec=5, max_retries=3, backoff=1.5):
    last_exc = None
    for attempt in range(1, max_retries + 1):
        try:
            timeout = aiohttp.ClientTimeout(total=timeout_sec)
            async with aiohttp.ClientSession(timeout=timeout) as sess:
                async with sess.post(callback_url, json=payload) as resp:
                    text = await resp.text()
                    if 200 <= resp.status < 300:
                        return {"status": resp.status, "body": text}
                    else:
                        last_exc = RuntimeError(f"bad status {resp.status}: {text}")
        except Exception as e:
            last_exc = e
        await asyncio.sleep(backoff ** attempt)
    raise last_exc

5) Make transcript save idempotent and fast (Python DB example)

async def save_transcript(transcript_data):
    # persist quickly so shutdown hook can re-send callback safely if needed
    await db.save_call_transcript(transcript_data)
    await db.update_call_status(transcript_data["call_id"], "completed", {
        "completed_at": datetime.datetime.now(), "transcript_saved": True
    })

6) Node.js (TS) equivalents

// server.ts
import { ServerOptions } from "livekit-agents"; // adjust import as needed

const opts = new ServerOptions({
  drainTimeout: 2400, // seconds
  // other options...
});

export default defineAgent({
  entry: async (ctx) => {
    ctx.addShutdownCallback(async () => {
      await finalizeTranscriptionAndSendCallback(callId, callbackUrl, transcriptData);
    });
    // job logic...
  },
});

HTTP callback with timeout+retry (Node.js using fetch + abort)

import fetch from "node-fetch";

async function sendCallbackNotification(url, payload, timeoutMs = 5000, retries = 3) {
  for (let i = 0; i < retries; i++) {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), timeoutMs);
    try {
      const res = await fetch(url, {
        method: "POST",
        body: JSON.stringify(payload),
        headers: { "Content-Type": "application/json" },
        signal: controller.signal,
      });
      clearTimeout(timer);
      if (res.ok) return await res.text();
      const text = await res.text();
      throw new Error(`HTTP ${res.status}: ${text}`);
    } catch (err) {
      if (i === retries - 1) throw err;
      await new Promise(r => setTimeout(r, 1000 * Math.pow(1.5, i)));
    }
  }
}

Notes / checklist • Register finalization using

ctx.add_shutdown_callback()

so the framework runs your hook during session shutdown. • Use

drain_timeout

to give jobs time to finish; LiveKit Cloud may have a max rollout window (up to 1 hour). • Keep callbacks short, add client timeouts, and retries so they don’t hang shutdown. • Persist transcript first, then send callback so retries are safe. If you want, tell me which exact file(s) you run (entrypoint name) and I’ll adapt the snippet into your agent’s code shape. Sources: Deployment reliability on LiveKit Cloud | LiveKit | Deployment reliability on LiveKit Cloud | LiveKit | Job lifecycle | LiveKit Documentation | Server options | LiveKit Documentation | Build a multilingual voice agent that automatically switches languages | Li..., with interactive elements

In which case livekit gracefully shutdown the session.. as i have updated the code but now i want to test it out.. then how can i test it properly..

Thanks — I’ll explain when LiveKit does a graceful shutdown and how you can test it end-to-end so your shutdown hook (save transcript + callback) runs reliably. When LiveKit performs a graceful shutdown • The agent receives SIGTERM or SIGINT (manual Ctrl+C, kill -TERM, or during rolling deploys). • The runtime stops accepting new jobs, marks worker unavailable, and enters drain mode. • Active jobs continue running; the agent waits up to the configured drain_timeout for them to finish. • After jobs complete (or drain_timeout expires) shutdown hooks registered with ctx.add_shutdown_callback() run, then the process exits. • Note: shutdown hooks have a short default wait (≈10s) unless you increase shutdown_process_timeout; LiveKit Cloud may cap graceful drain (up to ~1 hour). How to test graceful shutdown (practical steps) 1. Run your agent in start/production mode (not dev mode). 2. Register a shutdown hook that logs distinctly and awaits your final work (save + callback). Example log lines let you confirm the hook ran. 3. Trigger shutdown signals: ◦ Locally: send SIGTERM to the agent process:

kill -TERM <pid>

or press Ctrl+C. ◦ In container/K8s: delete the Pod or scale down so the platform sends SIGTERM. ◦ Simulate rolling deploy if you use LiveKit Cloud (old instance will receive SIGTERM). 1. Observe logs in this order: ◦ Worker becomes unavailable / no new jobs accepted ◦ Active job finishes (or drain begins) ◦ Your shutdown hook log appears (e.g., "Shutdown hook started", "Callback sent") ◦ Session report upload and process exit If you do not see your hook logs, the hook either didn’t run or was killed mid-run. 1. Shorten drain_timeout for fast tests: ◦ Set a small drain_timeout on your local AgentServer to speed iterations (e.g., 60s) so you can test behavior quickly. 1. Verify callback attempt details: ◦ Log callback request/response and errors inside send_callback_notification so you can tell whether the HTTP call was attempted and if it timed out or failed. Quick checklist of common issues to check while testing • Use exec in startup scripts so signals reach the process:

exec python agent.py

. • Ensure shutdown hook is registered with ctx.add_shutdown_callback(...) and all async tasks are awaited. • Give shutdown hooks enough time: increase

shutdown_process_timeout

and/or

drain_timeout

for tests. • Make the callback idempotent and persist transcript before attempting callback so retries are safe. • Add HTTP timeouts/retries to avoid hangs during shutdown. If you want, I can: • Provide a small test entrypoint you can run locally that logs each stage (enter, job end, shutdown hook start/finish), or • Generate a short K8s manifest showing how to send SIGTERM via Pod deletion for a real-world test. Which would you like me to give you next? Sources: Deployment reliability on LiveKit Cloud | LiveKit | Deployment reliability on LiveKit Cloud | LiveKit | Server startup modes | LiveKit Documentation | Job lifecycle | LiveKit Documentation | Why is my Agent insights data missing? | LiveKit, with interactive elements