bright-orange-86167
03/03/2026, 7:30 AMsleep(1s) in a loop) keeps ticking normally
- The exact same LiveKit Cloud server works from other clients
### What doesn't work
1. SDK internal 5s signal timeout — livekit_runtime::timeout() at
<http://signal_stream.rs:94|signal_stream.rs:94> never fires. The connection just hangs past 5s.
2. External tokio::time::timeout(30s) wrapping the spawned connect task —
also never fires.
3. std::thread watchdog (35s) — a dedicated OS thread that calls
std::thread::sleep(35s) then checks abort_handle.is_finished() and aborts
if still running. This ALSO doesn't trigger, which is puzzling since it's
completely independent of Tokio.
### Code structure
```rust
// Spawned in a separate task for cooperative cancellation
let connect_handle = tokio::spawn(async move {
Room::connect(&url, &token, room_options).await
});
let abort_handle = connect_handle.abort_handle();
// OS-thread watchdog (independent of Tokio timers)
let watchdog_abort = abort_handle.clone();
std:🧵:spawn(move || {
std:🧵:sleep(Duration::from_secs(35));
if !watchdog_abort.is_finished() {
watchdog_abort.abort();
warn!("watchdog triggered after 35s");
}
});
// Tokio timeout
match tokio:time:timeout(Duration::from_secs(30), connect_handle).await {
// ... handle results
}
Analysis
I traced through the SDK source and found that livekit-runtime re-exports
Tokio's sleep/timeout directly:
// livekit-runtime/src/tokio.rs
pub use tokio:time:sleep;
pub use tokio:time:timeout;
The connect path uses livekit_runtime::timeout() for signal connection (5s),
join response (5s), ICE connection (15s), etc. None of these fire.
My initial hypothesis was Tokio timer wheel waker loss on ARM/low-power hardware,
but the fact that the std::thread watchdog also fails to trigger suggests
something deeper — possibly the entire process or thread getting stuck in a
blocking syscall (TLS handshake? WebSocket upgrade?) that prevents both the
watchdog thread from completing its sleep AND the Tokio runtime from making progress.
Questions
1. Has anyone successfully run the Rust SDK on Raspberry Pi / ARM64?
2. Are there known issues with the signal stream WebSocket/TLS connection
blocking indefinitely on certain platforms?
3. Could the rustls + libwebrtc combination cause a blocking syscall
that starves both Tokio and OS threads?
4. Any recommended debugging steps to identify where exactly the connection
is stuck?
Workaround attempted
I replaced the tokio:time:timeout with a polling loop
(is_finished() + sleep(100ms)) — currently testing whether this makes
a difference. If the polling loop also hangs, it confirms the issue is not
timer-related but rather a blocking operation somewhere in the connect path.tall-airline-16756
03/03/2026, 7:31 AMtall-airline-16756
03/03/2026, 7:31 AMthread apply all bt full to get stack traces for all threads. Native frames inside libwebrtc, rustls, or libc will show where it’s stuck.
◦ If you see frames in OpenSSL/rustls/libwebrtc, that’s strong evidence the native layer is blocking.
1. Check dynamic libraries and symbol versions
◦ ldd target/debug/your_binary or ldd /proc/<pid>/exe — verify which TLS/crypto libs are used (system OpenSSL vs vendored).
◦ If webrtc or rustls forks are built against system libs, ABI or platform-specific mismatches can cause strange hangs.
1. Reproduce with minimal code path
◦ Create a tiny sample that only establishes the WSS connection used by SignalClient (bypass libwebrtc if possible) — e.g., use async-tungstenite or tokio-tls to open a WSS to your server using the same URL. If the minimal WSS client also hangs, the problem is the TLS/WebSocket stack or network path rather than LiveKit-specific code.
1. Exercise timeouts at different layers
◦ Replace livekit-runtime timeout with your own explicit task and use tokio::spawn_blocking for suspect blocking calls. If the blocking call runs inside spawn_blocking and still prevents the OS watchdog from waking, that’s an important clue (it shouldn’t; spawn_blocking moves work to a blocking pool).
1. Try a pure-std watchdog that examines process status
◦ Instead of simply sleeping, have the watchdog thread periodically read /proc/<pid>/stack or send SIGQUIT to dump backtraces if still hung. A SIGQUIT (kill -3) will cause Rust programs to print backtraces when enabled.
1. Enable extended logging and TLS debug
◦ Turn on debug logging for TLS or the WebSocket crate used by the signal client (if configurable). If rustls is used, enable its debug logging if available.
1. Test a different runtime / environment
◦ Try running the same binary on an x86 VM on the same network to separate network from platform issues.
◦ Try building with different feature flags: disable libwebrtc or use a different TLS backend if possible.
1. Try a polling workaround (what you started)
◦ Your polling approach (is_finished() + sleep(100ms)) is a good test. If that also hangs, the issue is not Tokio’s timer wheel. If the polling loop still advances, it indicates timers were the culprit.
1. Snapshot core dump for offline analysis
◦ If you can’t attach live, enable core dumps (ulimit -c unlimited) and collect core when it hangs. Analyze with gdb to inspect threads and native stacks.
Immediate practical checks I recommend you run first
• While the process is stuck, run: sudo cat /proc/<pid>/stack and sudo ls -l /proc/<pid>/fd
• Attach strace: sudo strace -ff -p <pid> -o /tmp/strace
• Attach gdb and run thread backtraces: sudo gdb -p <pid> then thread apply all bt
• Try your minimal WSS-only reproducer on the same device
What to look for in traces/backtraces
• Threads blocked in connect(), poll(), epoll_wait() are network I/O related.
• Threads blocked deep inside libwebrtc, OpenSSL, or rustls are TLS/crypto related.
• Futex or memory allocator deadlocks may show up as futex waits across threads.
• Frames showing syscall getrandom or RNG could indicate blocking during entropy gathering (rare on modern kernels but possible in early-boot or misconfigured systems).
If you capture traces or backtraces
• Share the thread backtraces (native frames) or the last strace syscalls. The key indicators are which syscall is blocked and which library function is on the stack (libwebrtc, rustls, openssl, libc).
Other notes and hypotheses
• The fact that std::thread sleep also didn’t wake suggests the process may be blocked in kernel context (syscall) or a low-level deadlock rather than just an async timer waker problem.
• libwebrtc contains native C++ code; if a blocking call happens there (for example certificate verification that calls into system libraries), it could stop progress and explain the symptoms.
• If your system uses some hardware crypto or odd kernel modules on ARM64, those can behave differently than x86 and cause platform-specific hangs.
If you want, I can:
• Walk through reading a sample strace or gdb backtrace you post and point to which library/function looks suspicious.
• Suggest exact strace/gdb commands to run tailored to your environment.
• Help craft a minimal WSS-only repro and a binary you can run to isolate signaling from full SDK/media stack.
Would you like the exact commands to attach gdb/strace and the most useful backtrace snippets to paste here? If so, tell me the PID when it’s hung (or paste a short strace/gdb output) and I’ll point out the relevant frames and next actions.
Sources: Rust SDKs | livekit/livekit_composite | DeepWiki | Handling asynchronous metadata updates in LiveKit | LiveKit | Firewall Tips for LiveKit Connectivity | LiveKit