This message was deleted.
# ask-ai
s
This message was deleted.
✅ 1
t
Thanks for reaching out, Abhishek. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I looked at the logs you pasted and the shutdown sequence they show. I’ll explain what happened, why the shutdown callback didn’t run, and next steps you can take to confirm and prevent it. What I’m addressing • Why the shutdown callback didn’t execute, based on the log timeline you shared for job_id AJ_aHkG2Df5XfPM (room RM_PVkea4QHX9r8). What the logs show (short timeline) • 103147 — session close events and connection resume attempts start. • 103158 — signal_event processing is slow handling an SDP Answer. • 103207 — “Subscriber pc state failed” and another session close. • 103211 — “process exiting” — graceful shutdown begins. • 103309 — “process did not exit in time, killing process” and a kill signal is sent. Why the shutdown callback didn’t execute • The process began a graceful shutdown but was forcibly killed before the shutdown callback could complete. • The logs show a forced termination after ~58 seconds of the process exiting. That indicates a shutdown timeout was reached and the orchestrator/framework sent a kill (SIGUSR1 and then kill) rather than waiting for your callback to finish. • Likely contributors visible in the logs: ◦ Ongoing WebRTC/connection issues (pc state failed, slow signal_event) left tasks blocked or unresponsive. ◦ A shutdown timeout (the process kill after “did not exit in time”) prevented the shutdown callback from running to completion. Concrete things to check now • Confirm what shutdown timeout your agent process or supervisor is configured with (the component that sends the kill after waiting). If it’s low (e.g., 10s), that can explain why callbacks are cut off. • Inspect the shutdown callback code for any blocking operations or long-running awaits (network I/O, unbounded waits). • Search logs for warnings or traces about hanging tasks or “non-daemon threads” during shutdown — those indicate what prevented exit. Suggested fixes to prevent future occurrences 1. Increase the process shutdown timeout so callbacks have time to complete. ◦ Raise the timeout used by your agent supervisor/process manager to a value that safely covers your cleanup logic. 1. Add timeouts and defensive handling inside your shutdown callback: ◦ Wrap long cleanup actions with an
asyncio.wait_for(..., timeout=...)
or equivalent so a single stuck cleanup won’t block everything. ◦ Catch and log exceptions/timeouts inside the callback so you can see what step timed out. 1. Make cleanup idempotent and quick: ◦ Close network connections and cancel background tasks early, then perform any slower cleanup asynchronously or off the hot path. 1. Add more shutdown logging: ◦ Log entry/exit for each step in your shutdown callback so you can pinpoint which step never completed next time. 1. Investigate the signal_event / pc_state failures: ◦ Those upstream WebRTC failures likely triggered the shutdown. Make sure your connection error handling doesn’t block shutdown (e.g., waiting indefinitely for remote responses). If you’d like, I can help with the next step: • Review the shutdown callback code (paste it here) and point out any blocking calls and where to add timeouts. • Identify the exact place your process manager sets the kill/timeout if you tell me how the process is launched or which server options you use. Would you like me to inspect the shutdown callback code or the process manager / server options next? Sources: Server options | LiveKit Documentation | livekit-agents/livekit/agents/worker.py
c
Can i retrive transcripts of this call from livekit?
t
Yes — you can retrieve transcripts from a LiveKit call, but how you get them depends on how transcription was set up during the call. Below are the possible methods and what to check for in your deployment. Which method applies to you? (quick question) • Was a LiveKit Agent with STT present in the room during that call? • If not, did you enable recording (egress) for that room and store recordings (S3/GCS)? If you tell me which of those is true, I can give exact retrieval steps. Meanwhile, here are the options: 1) Agent-based (real-time) transcription • If a LiveKit Agent with speech-to-text was in the room, the agent can emit final transcripts via events (for example
user_input_transcribed
) or write them to a file or external service. Check your agent logs or the agent’s storage location for saved transcript files. • If your agent was configured to forward transcripts as data messages / text streams (
lk.transcription
), clients may have received them in real time — check client-side logs or the server that the agent forwarded to. 2) Agent Observability (LiveKit Cloud) • If Agent Observability was enabled for the session, transcripts and session data may be stored and viewable/downloadable from the observability interface (Transcript tab with synchronized audio playback and logs). Check your LiveKit Cloud observability for that room/job id. 3) Recording (Egress) + external STT • If no agent was present, but you recorded the call via Universal Egress (track egress or other egress), you can: ◦ Retrieve the recorded audio file from your configured storage (S3/GCS). ◦ Run that audio through an STT service (e.g., Whisper, Google STT, Deepgram) to produce transcripts. Practical checks to run now • Look up the job_id (AJ_aHkG2Df5XfPM) / room_id (RM_PVkea4QHX9r8) in your agent logs and observability dashboard. • Check whether an agent process for that job emitted
user_input_transcribed
events or wrote transcript files. • Verify if any egress/recording was started for that room and where the output was stored. If you want, I can: • Walk you through retrieving transcripts from agent logs or the observability UI if you confirm Agent Observability was enabled. • Provide the exact steps to download egress files from your storage and run them through an STT pipeline if you recorded the call. • Review any agent code or config you paste here to point out where transcripts are written or forwarded. Which path should I help with — agent transcripts, observability lookup, or recorded-audio processing? Sources: Transcriber | LiveKit Documentation | Streamline troubleshooting with Agent Observability | LiveKit | Universal Egress | LiveKit | Text and transcriptions | LiveKit Documentation