<@U08CKJH7C12> seeing this error for multiple agen...
# agents
p
@refined-appointment-81829 seeing this error for multiple agent sessions - is anything down?
Copy code
{
  "message": "livekit_ffi::server::room:283:livekit_ffi::server::room - error while connecting to a room: engine: internal webrtc failure",
  "level": "ERROR",
  "name": "livekit",
  "pid": 286501,
  "job_id": "AJ_xbWkxnrQ3PQ7",
  "timestamp": "2025-11-19T18:07:17.770424+00:00"
}
Copy code
{
  "message": "livekit_ffi::server::room:283:livekit_ffi::server::room - error while connecting to a room: engine: internal webrtc failure",
  "level": "ERROR",
  "name": "livekit",
  "pid": 286501,
  "job_id": "AJ_xbWkxnrQ3PQ7",
  "timestamp": "2025-11-19T18:07:17.770424+00:00"
}
Copy code
{
  "message": "livekit_ffi::server::room:283:livekit_ffi::server::room - error while connecting to a room: engine: internal webrtc failure",
  "level": "ERROR",
  "name": "livekit",
  "pid": 286949,
  "job_id": "AJ_ydFStUBKk3Yb",
  "timestamp": "2025-11-19T18:11:18.262044+00:00"
}
Copy code
{
  "message": "livekit_ffi::server::room:283:livekit_ffi::server::room - error while connecting to a room: engine: internal webrtc failure",
  "level": "ERROR",
  "name": "livekit",
  "pid": 286985,
  "job_id": "AJ_Akj5XhP8hWBR",
  "timestamp": "2025-11-19T18:12:07.355595+00:00"
}
Copy code
{
  "message": "livekit_ffi::server::room:283:livekit_ffi::server::room - error while connecting to a room: engine: internal webrtc failure",
  "level": "ERROR",
  "name": "livekit",
  "pid": 287169,
  "job_id": "AJ_CMjGNAVDEkRc",
  "timestamp": "2025-11-19T18:12:43.627008+00:00"
}
Copy code
{
  "message": "livekit::rtc_engine:395:livekit::rtc_engine - failed to connect: Rtc(RtcError { error_type: Internal, message: \"failed to initialize pc\" }), retrying... (1/3)",
  "level": "WARNING",
  "name": "livekit",
  "pid": 286576,
  "job_id": "AJ_y2p8XxBwfmAm",
  "timestamp": "2025-11-19T18:07:54.462910+00:00"
}
This log is also present for all such cases
r
There is no outage
p
But the logs show that there is an
internal webrtc failure
. Can you please help check why that is the case? Can provide Room IDs if needed
I would really appreciate your help Wilson. This is affecting our prod env. And we are seeing multiple such cases. Project Id -
p_3tqm7ro6kbs
Here are the Room IDs where the above error has occurred: 1. RM_MJRxmCKxwAP2 2. RM_H732URyaisDA 3. RM_s5uLjy8dBvNp 4. RM_3vcD65LHqQ7R 5. RM_MJRxmCKxwAP2 6. RM_p4SkY5hroea6 I have ensured that the agent was ready to connect and was NOT resource constrained OR was not blocked by any other process - we don't run any blocking process at all before we connect to user and even on the main process and thread. Can you pls help check what happened here?
r
For
RM_MJRxmCKxwAP2
there was no worker available to take the job. Check your agent
p
The worker was definitely available at that time. I even received the job request
Copy code
{
  "message": "received job request",
  "level": "INFO",
  "name": "livekit.agents",
  "job_id": "AJ_xbWkxnrQ3PQ7",
  "dispatch_id": "AD_RZZ23PAv6LyC",
  "room_name": "tcn_stg_use2_000_inbound_prod_deployment_+17409191204_PfzHRt7VFWak",
  "agent_name": "prod_inbound",
  "resuming": false,
  "timestamp": "2025-11-19T18:07:17.355974+00:00"
}
The job didn't connect to the LiveKit room when called with
job.connect()
- At this step is where the above errors occurred
Copy code
{
  "levelname": "INFO",
  "name": "agent_orchestrator",
  "process": 286501,
  "event": "Waiting for job context to connect",
  "message": "",
  "pathname": "/home/app/agent-orchestrator/agent_orchestrator/service/call_service.py",
  "lineno": 1082,
  "call_id": "call_AJ_xbWkxnrQ3PQ7",
  "room_name": "tcn_stg_use2_000_inbound_prod_deployment_+17409191204_PfzHRt7VFWak",
  "tenant_id": "stg_use2_000",
  "timestamp": "2025-11-19T18:07:17.376723+00:00"
}
Nov 19 233717.494
Copy code
{
  "message": "livekit::rtc_engine:395:livekit::rtc_engine - failed to connect: Rtc(RtcError { error_type: Internal, message: \"failed to initialize pc\" }), retrying... (1/3)",
  "level": "WARNING",
  "name": "livekit",
  "pid": 286501,
  "job_id": "AJ_xbWkxnrQ3PQ7",
  "timestamp": "2025-11-19T18:07:17.494293+00:00"
}
r
invalid API key
We assigned the jon to the working but the API key the worker used was bad
p
We have the API key fetched from env variable and doesn't change at all. This is the
LIVEKIT_API_KEY
- env var. And at the same time other calls were running fine. Only the above 6 ran into this issue. Don't see any reason only this call's API key got corrupted
Cross posting the issue that I reported in a separate thread For the Room ID -
RM_JbRqV3rojmzp
. We are seeing the same
subscriber pc state failed
message and the process exits after retries along with the
signal_event taking too much time
Would appreciate your help here 🙏 https://livekit-users.slack.com/archives/C07FRP66XGE/p1764596112108939
r
What version of LiveKit are you running?
p
I am using livekit cloud with telephony Below are package versions I am using
Copy code
livekit==1.0.16
livekit-agents==1.2.17
livekit-api==1.0.7
livekit-blingfire==1.0.0
livekit-plugins-anthropic==1.2.17
livekit-plugins-cartesia==1.2.17
livekit-plugins-deepgram==1.2.17
livekit-plugins-elevenlabs==1.2.17
livekit-plugins-google==1.2.17
livekit-plugins-noise-cancellation==0.2.5
livekit-plugins-silero==1.2.17
livekit-plugins-turn-detector==1.2.17
livekit-protocol==1.0.4
👍🏼 1
One more call had the same issue again just now. Room ID -
RM_bG5kjdSwUCkJ
Copy code
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"server request to leave\" Migration Resume", "level": "WARNING", "name": "livekit", "pid": 33489, "job_id": "AJ_LKHJFyXLMtM6", "timestamp": "2025-12-01T16:08:24.324161+00:00"}
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"signal client closed: \\\"stream closed\\\"\" UnknownReason Resume", "level": "WARNING", "name": "livekit", "pid": 33489, "job_id": "AJ_LKHJFyXLMtM6", "timestamp": "2025-12-01T16:08:24.324489+00:00"}
{"message": "livekit::rtc_engine::rtc_session:639:livekit::rtc_engine::rtc_session - signal_event taking too much time: Answer(SessionDescription { r#type: \"answer\", sdp: \"v=0\\r\\no=- 1957616966143744551 1764605305 IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0 0\\r\\na=msid-semantic:WMS *\\r\\na=fingerprint:sha-256 0A:DC:C3:D6:57:5B:80:50:46:11:08:F0:25:07:75:EB:BA:4D:30:A0:EE:7E:D9:8E:30:47:DA:C9:55:EC:82:C5\\r\\na=ice-lite\\r\\na=extmap-allow-mixed\\r\\na=group:BUNDLE 0 1\\r\\nm=audio 9 UDP/TLS/RTP/SAVPF 63 111 0 8\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:active\\r\\na=mid:0\\r\\na=ice-ufrag:PiKwKfyNcGLFbXdo\\r\\na=ice-pwd:gnHbAWsQVWktvDsfWFZYqlWEhmwZrQRB\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=rtpmap:63 red/48000/2\\r\\na=fmtp:63 111/111\\r\\na=rtpmap:111 opus/48000/2\\r\\na=fmtp:111 minptime=10;useinbandfec=1;usedtx=1\\r\\na=rtpmap:0 PCMU/8000\\r\\na=rtpmap:8 PCMA/8000\\r\\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\\r\\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\\r\\na=recvonly\\r\\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:active\\r\\na=mid:1\\r\\na=sendrecv\\r\\na=sctp-port:5000\\r\\na=max-message-size:65535\\r\\na=ice-ufrag:PiKwKfyNcGLFbXdo\\r\\na=ice-pwd:gnHbAWsQVWktvDsfWFZYqlWEhmwZrQRB\\r\\n\", id: 0 })", "level": "ERROR", "name": "livekit", "pid": 33489, "job_id": "AJ_LKHJFyXLMtM6", "timestamp": "2025-12-01T16:08:35.380051+00:00"}
r
What percentage of calls does this happen?
p
This is randomly happening for 3% to 5% calls daly.
👍🏼 1
r
Who is the SIP provider? Looks like maybe something could be wrong with that SIP service.
p
SIP Provider is TCN
Are there any more details on what is wrong in the SIP provider side?
r
I am seeing media errors. I am checking with the team to get a second opinion
p
Thank you so much
Hello CWilson - is there any update on the issue here please?
r
No, I did not hear anything. I will try and follow up with team again.
🙏 1
So the issue you report above about
Migration Resume
what problem did you have with that? The client should have reconnected automatically. Was there some issue?
p
Yes the reconnection never occurs - the reconnection fails with
pc state failed
Do I have to upgrade to some different library version to ensure reconnection happens?
r
Running newer version could help
p
Newer versions of all the libraries OR can we only upgrade certain libraries that perform this retry? Would really appreciate your help here. What I am meaning to ask is: 1. Which library performs this retry that I can upgrade 2. Can I upgrade the library without affecting other libraries like agents framework which has some of our custom code and needs to be migrated to 3.0 version
@refined-appointment-81829 - apologies for pinging again, is it possible to answer the above qs?
r
Did you get a chance to test without modified agents and code to see if you can reproduce?
p
without modified agents and code
Sorry did not understand clearly
r
I think the issue maybe related to changes that have been made to the code. Try and reproduce with released version of the code and see if the issue goes away.
p
I don't have reproducible steps. I don't even know why the participant migration happens in the first place.
Try and reproduce with released version of the code and see if the issue goes away.
By released version - which library and version - is it the livekit-rtc package with the latest version OR livekit-agents with the latest version Apologies for a lot of questions, it is just that I am consfused on the responses
r
I mean the full LiveKit stack
p
Can you help me understand what does
Participant Migration
mean - all the failure cases have this in common. I will see how quickly I can upgrade all the libraries of livekit. I am already using Livekit cloud.
@refined-appointment-81829 - this has appeared again 😞 . Can you pls help what could be the issue here? Room ID -
RM_RVDH2hbjYw7Z
(Don't see complete info on Dashboard yet) Project ID -
p_3tqm7ro6kbs
Copy code
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"server request to leave\" Migration Resume", "level": "WARNING", "name": "livekit", "pid": 371890, "job_id": "AJ_9oSCDKgqJWLU", "timestamp": "2025-12-17T17:46:15.082028+00:00"}
Nearby Logs
Dec 17 23:16:15.082
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"signal client closed: \\\"stream closed\\\"\" UnknownReason Resume", "level": "WARNING", "name": "livekit", "pid": 371890, "job_id": "AJ_9oSCDKgqJWLU", "timestamp": "2025-12-17T17:46:15.082151+00:00"}
Copy code
{
  "message": "livekit::rtc_engine::rtc_session:639:livekit::rtc_engine::rtc_session - signal_event taking too much time: Answer(SessionDescription { r#type: \"answer\", sdp: \"v=0\\r\\no=- 7773977208008433610 1765993576 IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0 0\\r\\na=msid-semantic:WMS *\\r\\na=fingerprint:sha-256 9F:44:70:B9:23:E6:6A:7C:B3:BF:13:09:F0:D5:E0:73:F1:E8:5D:BB:9C:80:1F:8C:AC:22:DD:70:C2:19:5E:B6\\r\\na=ice-lite\\r\\na=extmap-allow-mixed\\r\\na=group:BUNDLE 0 1\\r\\nm=audio 9 UDP/TLS/RTP/SAVPF 63 111 0 8\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:active\\r\\na=mid:0\\r\\na=ice-ufrag:yfmhoBwacKYPEeED\\r\\na=ice-pwd:WqxLGFRDtjSuucVNlplrlClxbMrsRJwr\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=rtpmap:63 red/48000/2\\r\\na=fmtp:63 111/111\\r\\na=rtpmap:111 opus/48000/2\\r\\na=fmtp:111 minptime=10;useinbandfec=1;usedtx=1\\r\\na=rtpmap:0 PCMU/8000\\r\\na=rtpmap:8 PCMA/8000\\r\\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\\r\\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\\r\\na=recvonly\\r\\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:active\\r\\na=mid:1\\r\\na=sendrecv\\r\\na=sctp-port:5000\\r\\na=max-message-size:65535\\r\\na=ice-ufrag:yfmhoBwacKYPEeED\\r\\na=ice-pwd:WqxLGFRDtjSuucVNlplrlClxbMrsRJwr\\r\\n\", id: 0 })",
  "level": "ERROR",
  "name": "livekit",
  "pid": 371890,
  "job_id": "AJ_9oSCDKgqJWLU",
  "timestamp": "2025-12-17T17:46:26.157923+00:00"
}
Copy code
{"message": "livekit::rtc_engine::rtc_session:960:livekit::rtc_engine::rtc_session - Subscriber pc state failed", "level": "ERROR", "name": "livekit", "pid": 371890, "job_id": "AJ_9oSCDKgqJWLU", "timestamp": "2025-12-17T17:46:34.967359+00:00"}
Dec 17 23:16:34.968
{"message": "livekit::rtc_engine:453:livekit::rtc_engine - received session close: \"pc_state failed\" UnknownReason Resume", "level": "WARNING", "name": "livekit", "pid": 371890, "job_id": "AJ_9oSCDKgqJWLU", "timestamp": "2025-12-17T17:46:34.967550+00:00"}
Dec 17 23:16:34.968
{"message": "livekit::rtc_engine:729:livekit::rtc_engine - resuming connection... attempt: 0", "level": "ERROR", "name": "livekit", "pid": 371890, "job_id": "AJ_9oSCDKgqJWLU", "timestamp": "2025-12-17T17:46:34.967655+00:00"}
Dec 17 23:16:35.066
{"message": "livekit::rtc_engine::rtc_session:794:livekit::rtc_engine::rtc_session - Wrong packet sequence while retrying: 220 > 215, 5 packets missing", "level": "WARNING", "name": "livekit", "pid": 371890, "job_id": "AJ_9oSCDKgqJWLU", "timestamp": "2025-12-17T17:46:35.066540+00:00"}
I see a new event called
Participant left (STATE_MISMATCH)
and I am still unaware what
Participant left (MIGRATION)
is about. Can you please help answer this 🙏 - would appreciate your response
l
Hi Zaheer, I'll try and summarize because I'm a bit confused when I went through this thread, but here:
1. Can I upgrade the library without affecting other libraries like agents framework which has some of our custom code and needs to be migrated to 3.0 version
You spoke about having made custom changes to the agents framework - obviously LiveKit is open source, so there's nothing stopping you doing that, but it does make it more tricky to support and provide guidance. CWilson was asking if you also saw this issue with the released version of LiveKit, without your custom changes. I would also have the same question because I don't see any other issues or recent discussions related to
MIGRATION
. Until we can reproduce this with the released code, I think we would struggle to isolate the issue. Regarding what those events actually do, as far as I'm aware it is related to the load balancer for LiveKit cloud, but I don't see any documentation for it. The best resource in this case would be the source code, and I found this helpful: https://deepwiki.com/search/what-can-cause-the-event-parti_fd17518b-d5c0-4d34-b982-10ddb832a23b?mode=fast
p
Thank you for the response Darryn. Apologies if my behavior came off as rude. I understand that we need to upgrade this and will be upgrading soon. I just wanted to know why this is occurring specifically for older sdk versions as this was NOT mentioned in the release that this might have been fixed. So just wanted to understand the root cause. And NOT denying the fact that we are on a stale version. Will look to upgrade asap on this. Appreciate your help 🙏
l
I didn't think your behaviour came off as rude at all 🙂 I'm glad we are on the same page. Best case scenario, this issue goes away when you upgrade and add your custom changes back in.
I just wanted to know why this is occurring specifically for older sdk versions
It's not just older SDK versions, as we haven't seen this issue before I don't believe... it's more a concern that your custom changes are having some unknown side effect that might be triggering something. Hence why it would be good to know if you see this behaviour after upgrading to the latest base version, then after adding your custom changes back in.