https://livekit.io logo
Join Slack
Powered by
Hi team, we seem to be facing lots of instability ...
# cloud
f
Hi team, we seem to be facing lots of instability from LiveKit cloud as a transport layer when running jobs with Modal.
Copy code
Connecting to room: voice-1767066295
[2025-12-30 03:45:45] INFO _ffi_client.py:167: livekit_ffi::server:139:livekit_ffi::server - initializing ffi server v0.12.42
[2025-12-30 03:45:45] INFO _ffi_client.py:167: livekit_ffi::cabi:50:livekit_ffi::cabi - initializing ffi server v0.12.42
Nvidia Encoder is supported.
Nvidia Decoder is supported.
[2025-12-30 03:45:45] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:46:00] WARNING _ffi_client.py:167: livekit::rtc_engine:405:livekit::rtc_engine - failed to connect: Connection("wait_pc_connection timed out"), retrying... (1/3)
[2025-12-30 03:46:00] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
Connected! Room: voice-1767066295, Local participant: voice-agent
Subscribed to audio from user-1304
Failed to send: engine: connection error: could not establish publisher connection: timeout
[2025-12-30 03:46:16] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1772:livekit::rtc_engine::rtc_session - connection error: could not establish publisher connection: timeout
[2025-12-30 03:46:16] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1041:livekit::rtc_engine::rtc_session - Publisher pc state failed
[2025-12-30 03:46:16] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "pc_state failed" UnknownReason Resume
[2025-12-30 03:46:16] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "server request to leave" ConnectionTimeout Resume
[2025-12-30 03:46:16] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "signal client closed: \"stream closed\"" UnknownReason Resume
[2025-12-30 03:46:22] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1041:livekit::rtc_engine::rtc_session - Subscriber pc state failed
[2025-12-30 03:46:22] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "pc_state failed" UnknownReason Resume
[2025-12-30 03:46:31] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1772:livekit::rtc_engine::rtc_session - connection error: could not establish publisher connection: timeout
[2025-12-30 03:46:31] ERROR _ffi_client.py:167: livekit::rtc_engine:773:livekit::rtc_engine - resuming connection... attempt: 0
[2025-12-30 03:46:31] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23&reconnect=1&sid=PA_MY2cuzU9piGK>
Failed to send: engine: connection error: could not establish publisher connection: timeout
[2025-12-30 03:46:31] ERROR _ffi_client.py:167: livekit::rtc_engine:775:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally
[2025-12-30 03:46:31] ERROR _ffi_client.py:167: livekit::rtc_engine:749:livekit::rtc_engine - restarting connection... attempt: 1
[2025-12-30 03:46:31] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:46:46] ERROR _ffi_client.py:167: livekit::rtc_engine:759:livekit::rtc_engine - restarting connection failed: connection error: wait_pc_connection timed out
[2025-12-30 03:46:46] ERROR _ffi_client.py:167: livekit::rtc_engine:749:livekit::rtc_engine - restarting connection... attempt: 2
[2025-12-30 03:46:46] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:47:02] ERROR _ffi_client.py:167: livekit::rtc_engine:759:livekit::rtc_engine - restarting connection failed: connection error: wait_pc_connection timed out
[2025-12-30 03:47:02] ERROR _ffi_client.py:167: livekit::rtc_engine:749:livekit::rtc_engine - restarting connection... attempt: 3
[2025-12-30 03:47:02] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:47:17] ERROR _ffi_client.py:167: livekit::rtc_engine:759:livekit::rtc_engine - restarting connection failed: connection error: wait_pc_connection timed out
[2025-12-30 03:47:17] ERROR _ffi_client.py:167: livekit::rtc_engine:749:livekit::rtc_engine - restarting connection... attempt: 4
[2025-12-30 03:47:17] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:47:32] ERROR _ffi_client.py:167: livekit::rtc_engine:759:livekit::rtc_engine - restarting connection failed: connection error: wait_pc_connection timed out
[2025-12-30 03:47:32] ERROR _ffi_client.py:167: livekit::rtc_engine:749:livekit::rtc_engine - restarting connection... attempt: 5
[2025-12-30 03:47:32] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:47:32] INFO _ffi_client.py:167: livekit::rtc_engine:702:livekit::rtc_engine - RtcEngine successfully recovered
Subscribed to audio from user-1304
Failed to send: engine: connection error: could not establish publisher connection: timeout
[2025-12-30 03:47:47] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1772:livekit::rtc_engine::rtc_session - connection error: could not establish publisher connection: timeout
[2025-12-30 03:47:47] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1041:livekit::rtc_engine::rtc_session - Publisher pc state failed
[2025-12-30 03:47:47] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "pc_state failed" UnknownReason Resume
[2025-12-30 03:47:47] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "server request to leave" ConnectionTimeout Resume
[2025-12-30 03:47:47] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "signal client closed: \"stream closed\"" UnknownReason Resume
[2025-12-30 03:47:53] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1041:livekit::rtc_engine::rtc_session - Subscriber pc state failed
[2025-12-30 03:47:53] WARNING _ffi_client.py:167: livekit::rtc_engine:474:livekit::rtc_engine - received session close: "pc_state failed" UnknownReason Resume
[2025-12-30 03:48:02] ERROR _ffi_client.py:167: livekit::rtc_engine::rtc_session:1772:livekit::rtc_engine::rtc_session - connection error: could not establish publisher connection: timeout
Failed to send: engine: connection error: could not establish publisher connection: timeout
[2025-12-30 03:48:02] ERROR _ffi_client.py:167: livekit::rtc_engine:773:livekit::rtc_engine - resuming connection... attempt: 0
[2025-12-30 03:48:02] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23&reconnect=1&sid=PA_ZhtQZZ5R45xd>
[2025-12-30 03:48:02] ERROR _ffi_client.py:167: livekit::rtc_engine:775:livekit::rtc_engine - resuming connection failed: signal failure: ws failure: Connection closed normally
[2025-12-30 03:48:02] ERROR _ffi_client.py:167: livekit::rtc_engine:749:livekit::rtc_engine - restarting connection... attempt: 1
[2025-12-30 03:48:02] INFO _ffi_client.py:167: livekit_api::signal_client::signal_stream:92:livekit_api::signal_client::signal_stream - connecting to <wss://testing-sevtctxq.livekit.cloud/rtc?sdk=python&protocol=16&auto_subscribe=1&adaptive_stream=0&version=1.0.23>
[2025-12-30 03:48:03] INFO _ffi_client.py:167: livekit::rtc_engine:702:livekit::rtc_engine - RtcEngine successfully recovered
Subscribed to audio from user-1304
Modal logo
© 2025
About
Status
Changelog
Our workers will repeatedly fail to connect to livekit and hang as a result.
m
You're hosting agents on Modal?
f
Yes, we're experimenting with something cool where we need colo GPUs. I'll send you a demo when it's up and running 🙂
👍 1
(None of this is prod btw, just the team tinkering)
m
I do think there's a known issue on Modal’s side with hosting LK agents on it — let me check internally
🙌 1
✅ 1
f
It's weird cause it sometimes works. Maybe 20% connect rate.
g
I'm experiencing a strange issue. • I have a nextjs / react livekit client hosted on cloudflare • We are deployed on livekit agents. Under certain conditions connects and then IMMEDIATELY disconnects. • Locally this happens 100% of the time. • On the production website it happens 0 % of the time. However, users are saying that is connects or does not connect based on whether they are on mobile / WiFi. Do you all know how I debug this? Or what may be happening
r
What do you mean by: > We are deployed on livekit agents.
Also what does this mean? > Locally this happens 100% of the time.
g
Ah yes! This is my first time deploying on livekit hosted agents
I have traditionally just used a VM
Locally this happens 100% of the time.
A little bit strange but when I just the nextjs app locally it immediately connects and then disconnects. That same commit hash deployed behaves fine
I'm not sure if it's a localhost issue, I've never had something like happen before
Are there certain logs I can focus on to see what is happening? is likely a TURN server / reconnection issue?
r
When you say NextJS app you mean the client connects then disconnects. Even if the Agent is not in the room?
g
Yes, I believe I share the backend logs and frontend logs
r
g
👀
I think what is suprising to me is it's the same browser
r
You have a session ID where the "local" NextJS app connects then disconnects
g
yes! and I'm connecting to the sdame deployed agent
on prod and on local
I see in the backend logs "client has disconnected" within 1-2 seconds of establishing connection
Is this helpful?
r
Do you see anything the the browsers JS console?
g
Forgive me the noise I was desperate and trying to bombard claude code
I tried staring at the logs, but nothing out of the ordinary cuaght my eye
I think 
connection quality excelllent
is what confuses me a bit
Also forgive I was just trying to configure loguru and have an error in my log formatting
But I always see this after three seconds:
Copy code
Room connection state changed | state=0[22:05:13] INFO: Room disconnected[22:05:13] INFO: Waiting for potential reconnection | grace_seconds=10[22:05:13] (livekit.agents/55): process exiting^[[O^[[I
r
Is it possible two clients are trying to use the same token?
Or same name. This might cause what you are seeing when the second one joins the first will be booted
g
I'm not sure to be honest! The confusing part for me is prod is super seamless, 500 ms and I hear TTS!
Is it possible two clients are trying to use the same token?
[10:12 PM]
Or same name. This might cause what you are seeing when the second one joins the first will be booted
👀
I
I hope it's something like this:
would make sense of why local react had it
I'll disable strict mode and nuke the dev server
ah okay no dice, claude code use a set to prevent the same from being used twice
I dont actually mind testing w/ prod agent right now, but just not sure if its a wider connection issue or just some weird dev env / localhost issue
Not sure what on the client is causing a dis connect
r
Can you go here and provide a session id for a failure case: https://cloud.livekit.io/projects/p_/sessions
I looked at this one I saw in your screen capture but the client requested to leave room: 
RM_ihr3HPiUkL7y
g
Copy code
RM_pQr6J7Up86i6
Is there a client disconnect hook I can debug on?
wonder if the react-sdk records who / why requested a disconnect
I
I'm like 95% sure the server is happy and something very weird is happening on the client
r
Pretty sure it is something in your client app. I see 
CLIENT_REQUEST_LEAVE
from that user.
Nothing looks out of the ordinary on the server logs
Maybe it is in dev mode and something is causing a view reload? That might do this
3 Views
Open in Slack
PreviousNext
Powered by