Seeing the below error messages. Anyone else also ...
# agents
p
Seeing the below error messages. Anyone else also seeing this? Why does this timeout occur? Is it between the Agent process and LiveKit cloud?
Copy code
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"signal client closed: \\\"ping timeout\\\"\" UnknownReason Resume", "level": "WARNING", "name": "livekit", "pid": 77515, "job_id": "AJ_MhbAELG4WnuH", "timestamp": "2025-11-13T15:10:53.015125+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 0", "level": "ERROR", "name": "livekit", "pid": 77515, "job_id": "AJ_MhbAELG4WnuH", "timestamp": "2025-11-13T15:10:53.016054+00:00"}
cc: @refined-appointment-81829
r
I am not sure what is going on there. Looks like something wrong with the SIP provider. Who are you using for SIP. Maybe they have network issues?
p
Wilson - this was in the Agent's process that was resuming due to ping timeout not related to SIP I believe (?) Does LiveKit cloud close the session if there is any ping timeout?
@jolly-air-155 - would you be able to help here? Don't have replication steps - but this occurs randomly. Want to know why ping timeout occurs. What should I check on our end to ensure ping timeouts do not occur. Is there a way to configure the ping timeouts Also ran into a race condition where the livekit room is being resumed during session closing causing audio output to be lost and raising an exception in local track
Copy code
{
  "message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"signal client closed: \\\"ping timeout\\\"\" UnknownReason Resume",
  "level": "WARNING",
  "name": "livekit",
  "pid": 152786,
  "job_id": "AJ_fyxVgrKTusfQ",
  "timestamp": "2025-11-17T16:46:46.741406+00:00"
}
Copy code
{
  "message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 0",
  "level": "ERROR",
  "name": "livekit",
  "pid": 152786,
  "job_id": "AJ_fyxVgrKTusfQ",
  "timestamp": "2025-11-17T16:46:46.742088+00:00"
}
Copy code
{"message": "error running user callback for local_track_published: room_handle: 4\nlocal_track_published {\n  track_sid: \"TR_AMe7snshHNrUpa\"\n}\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/room.py\", line 560, in _listen_task\n    self._on_room_event(event.room_event)\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/room.py\", line 607, in _on_room_event\n    lpublication = self.local_participant.track_publications[sid]\n                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^\nKeyError: 'TR_AMe7snshHNrUpa'", "level": "ERROR", "name": "root", "pid": 152786, "job_id": "AJ_fyxVgrKTusfQ", "timestamp": "2025-11-17T16:47:11.333984+00:00"}
Nearby Logs
Nov 17 22:17:11.336
{"message": "error running user callback for local_track_subscribed: room_handle: 4\nlocal_track_subscribed {\n  track_sid: \"TR_AMe7snshHNrUpa\"\n}\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/room.py\", line 560, in _listen_task\n    self._on_room_event(event.room_event)\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/room.py\", line 616, in _on_room_event\n    lpublication = self.local_participant.track_publications[sid]\n                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^\nKeyError: 'TR_AMe7snshHNrUpa'", "level": "ERROR", "name": "root", "pid": 152786, "job_id": "AJ_fyxVgrKTusfQ", "timestamp": "2025-11-17T16:47:11.336074+00:00"}
r
This looks like a network issue.
p
Network failures from our side to livekit cloud?
r
your side
👍 1
p
@refined-appointment-81829 - for the below call the call -
AJ_Q5uCDLQoKZpu
. The reconnection to LiveKit wasn't successful. Room ID -
RM_mhRYGQPWssBb
. It tried resuming the conenction but it didn't go through, can you please help check why this can happen? This has caused an issue where the agent was stuck and wasn't responding for 7 mins The new room session it was trying to connect to -
RM_VujmsKk4v6kF
.
Copy code
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"signal client closed: \\\"ping timeout\\\"\" UnknownReason Resume", "level": "WARNING", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T15:57:48.870258+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 0", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T15:57:48.870746+00:00"}
{"message": "_SegmentSynchronizerImpl.resume called after close", "level": "WARNING", "name": "livekit.agents", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T15:58:02.436703+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.240043+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 1", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.240288+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.332808+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 2", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.332983+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.422920+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 3", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.423081+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.515362+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 4", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.515510+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.607453+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 5", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.608397+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.698840+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 6", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.699109+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.791497+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 7", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.791678+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.881934+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 8", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.882097+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.975011+00:00"}
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 9", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:33.975199+00:00"}
{"message": "livekit::rtc_engine:731:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.065930+00:00"}
{"message": "livekit::rtc_engine:663:livekit::rtc_engine - failed to reconnect to the livekit room", "level": "ERROR", "name": "livekit", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.066113+00:00"}
{"message": "failed to publish transcription\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/agents/voice/room_io/_output.py\", line 431, in capture_text\n    await self._writer.write(text)\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/data_stream.py\", line 295, in write\n    await self._send_chunk(chunk_msg)\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/data_stream.py\", line 213, in _send_chunk\n    raise ConnectionError(cb.send_stream_chunk.error)\nConnectionError: engine: connection error: engine is closed", "level": "WARNING", "name": "livekit.agents", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.067520+00:00"}
{"message": "failed to publish transcription\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/agents/voice/room_io/_output.py\", line 300, in _publish_transcription\n    await self._room.local_participant.publish_transcription(transcription)\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/participant.py\", line 285, in publish_transcription\n    raise PublishTranscriptionError(cb.publish_transcription.error)\nlivekit.rtc.participant.PublishTranscriptionError: engine: connection error: engine is closed", "level": "WARNING", "name": "livekit.agents", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.071639+00:00"}
{"message": "failed to publish transcription\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/agents/voice/room_io/_output.py\", line 300, in _publish_transcription\n    await self._room.local_participant.publish_transcription(transcription)\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/participant.py\", line 285, in publish_transcription\n    raise PublishTranscriptionError(cb.publish_transcription.error)\nlivekit.rtc.participant.PublishTranscriptionError: engine: connection error: engine is closed", "level": "WARNING", "name": "livekit.agents", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.073389+00:00"}
{"levelname": "INFO", "name": "agent_orchestrator", "process": 272419, "event": "Shutdown Call", "message": "", "pathname": "/home/app/agent-orchestrator/agent_orchestrator/service/call_service.py", "lineno": 1551, "end_reason": "room disconnected", "call_id": "call_AJ_Q5uCDLQoKZpu", "room_name": "tcn_stg_use2_000_inbound_prod_deployment_+15749036527_ptWYTQRqhzLU", "tenant_id": "stg_use2_000", "timestamp": "2025-11-19T16:03:34.074218+00:00"}
{"message": "process exiting", "level": "INFO", "name": "livekit.agents", "reason": "room disconnected", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.074274+00:00"}
{"levelname": "INFO", "name": "agent_orchestrator", "process": 272419, "event": "Shutdown Call - Call is Ongoing - Triggering Call End", "end_reason": "room disconnected", "timestamp": "2025-11-19T16:03:34.074358+00:00"}
{"levelname": "INFO", "name": "agent_orchestrator", "process": 272419, "event": "End Call", "end_reason": "room disconnected",  "timestamp": "2025-11-19T16:03:34.074457+00:00"}
{"message": "failed to publish transcription\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/livekit/agents/voice/room_io/_output.py\", line 433, in capture_text\n    tmp_writer = await self._create_text_writer()\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/livekit/agents/voice/room_io/_output.py\", line 405, in _create_text_writer\n    return await self._room.local_participant.stream_text(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/participant.py\", line 578, in stream_text\n    await writer._send_header()\n  File \"/usr/local/lib/python3.11/site-packages/livekit/rtc/data_stream.py\", line 189, in _send_header\n    raise ConnectionError(cb.send_stream_header.error)\nConnectionError: engine: connection error: engine is closed", "level": "WARNING", "name": "livekit.agents", "pid": 272419, "job_id": "AJ_Q5uCDLQoKZpu", "timestamp": "2025-11-19T16:03:34.073676+00:00"}
{"levelname": "INFO", "name": "agent_orchestrator", "process": 272419, "event": "Deleting Room", "message": "", "pathname": "/home/app/agent-orchestrator/agent_orchestrator/service/call_service.py", "lineno": 2285, "timestamp": "2025-11-19T16:03:38.277132+00:00"}
{"levelname": "ERROR", "name": "agent_orchestrator", "process": 272419, "event": "Delete Room Error", "message": "", "pathname": "/home/app/agent-orchestrator/agent_orchestrator/service/call_service.py", "lineno": 2307, "error": "TwirpError(code=not_found, message=requested room does not exist, status=404)", "exc_info": "Traceback (most recent call last):\n  File \"/home/app/agent-orchestrator/agent_orchestrator/service/call_service.py\", line 2293, in delete_room\n    await self.job_ctx.api.room.delete_room(\n  File \"/usr/local/lib/python3.11/site-packages/livekit/api/room_service.py\", line 108, in delete_room\n    return await self._client.request(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/livekit/api/twirp_client.py\", line 129, in request\n    raise TwirpError(\nlivekit.api.twirp_client.TwirpError: TwirpError(code=not_found, message=requested room does not exist, status=404)", "timestamp": "2025-11-19T16:03:38.336024+00:00"}