Room::connect hangs indefinitely on Raspberry Pi 5...
# ask-ai
b
Room::connect hangs indefinitely on Raspberry Pi 5 (ARM64) — internal SDK timeouts never fire I'm running the Rust LiveKit SDK (livekit 0.7.28, livekit-runtime 0.4.0) on a Raspberry Pi 5 (ARM64, Debian Bookworm) and experiencing Room::connect hanging indefinitely during the WSS/signal connection phase. ### Environment - Raspberry Pi 5 (ARM64, 8GB) - Debian Bookworm (Linux 6.12) - Tokio runtime with multi-thread feature - livekit = "0.7", livekit-api = "0.4", libwebrtc = "0.3" - LiveKit Cloud server (wss://xxx.livekit.cloud) ### Symptoms Room::connect hangs at the signal stream connection step and never returns. The log stops at: INFO [livekit_api:signal client:signal_stream] connecting to wss://xxx.livekit.cloud/rtc?sdk=rust&protocol=16&... After this line, nothing happens — no success, no error, no timeout. ### What works - DNS resolution succeeds (IPv4 addresses returned) - Token fetch works fine - Tokio heartbeat task (
sleep(1s)
in a loop) keeps ticking normally - The exact same LiveKit Cloud server works from other clients ### What doesn't work 1. SDK internal 5s signal timeout —
livekit_runtime::timeout()
at
<http://signal_stream.rs:94|signal_stream.rs:94>
never fires. The connection just hangs past 5s. 2. External
tokio::time::timeout(30s)
wrapping the spawned connect task — also never fires. 3.
std::thread
watchdog (35s)
— a dedicated OS thread that calls
std::thread::sleep(35s)
then checks
abort_handle.is_finished()
and aborts if still running. This ALSO doesn't trigger, which is puzzling since it's completely independent of Tokio. ### Code structure ```rust // Spawned in a separate task for cooperative cancellation let connect_handle = tokio::spawn(async move { Room::connect(&url, &token, room_options).await }); let abort_handle = connect_handle.abort_handle(); // OS-thread watchdog (independent of Tokio timers) let watchdog_abort = abort_handle.clone(); std:🧵:spawn(move || { std:🧵:sleep(Duration::from_secs(35)); if !watchdog_abort.is_finished() { watchdog_abort.abort(); warn!("watchdog triggered after 35s"); } }); // Tokio timeout match tokio:time:timeout(Duration::from_secs(30), connect_handle).await { // ... handle results } Analysis I traced through the SDK source and found that livekit-runtime re-exports Tokio's sleep/timeout directly: // livekit-runtime/src/tokio.rs pub use tokio:time:sleep; pub use tokio:time:timeout; The connect path uses livekit_runtime::timeout() for signal connection (5s), join response (5s), ICE connection (15s), etc. None of these fire. My initial hypothesis was Tokio timer wheel waker loss on ARM/low-power hardware, but the fact that the std::thread watchdog also fails to trigger suggests something deeper — possibly the entire process or thread getting stuck in a blocking syscall (TLS handshake? WebSocket upgrade?) that prevents both the watchdog thread from completing its sleep AND the Tokio runtime from making progress. Questions 1. Has anyone successfully run the Rust SDK on Raspberry Pi / ARM64? 2. Are there known issues with the signal stream WebSocket/TLS connection blocking indefinitely on certain platforms? 3. Could the rustls + libwebrtc combination cause a blocking syscall that starves both Tokio and OS threads? 4. Any recommended debugging steps to identify where exactly the connection is stuck? Workaround attempted I replaced the tokio:time:timeout with a polling loop (is_finished() + sleep(100ms)) — currently testing whether this makes a difference. If the polling loop also hangs, it confirms the issue is not timer-related but rather a blocking operation somewhere in the connect path.
t
Thanks for reaching out, 하대훈. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is a clear, well-instrumented report and I can see why it’s puzzling: timeouts inside the SDK, the Tokio timeout, and even a plain OS-thread watchdog all never fire. I’ll respond to each of your numbered questions and then give targeted debugging steps that should help localize whether this is an async runtime/timer issue, a blocking syscall in native code, or something else. Quick summary of what I’m addressing • You asked whether the Rust SDK is known to run on Raspberry Pi/ARM64, whether signal WebSocket/TLS can block indefinitely on some platforms, whether rustls + libwebrtc could cause blocking syscalls that stall both Tokio and OS threads, and what debugging steps to take. Short answers to your numbered questions 1) Has anyone successfully run the Rust SDK on Raspberry Pi / ARM64? • The SDK supports aarch64 Linux targets and there are prebuilt libraries for aarch64; running on ARM64 Linux is supported in principle. 2) Are there known issues with the signal stream WebSocket/TLS connection blocking indefinitely on certain platforms? • There is no documented built-in fallback for a stalled signaling WebSocket connection. A hang during the WebSocket/TLS handshake is possible if a blocking syscall occurs in native layers, which can make the connect attempt appear to hang indefinitely. 3) Could rustls + libwebrtc cause a blocking syscall that starves both Tokio and OS threads? • Yes. If native code (libwebrtc or TLS integration) performs a blocking syscall or enters a deadlock in the process, it can prevent progress across threads and async timers. That matches your symptom set where both Tokio timers and std::thread sleeps never complete. 4) Recommended debugging steps to identify where the connection is stuck • I list prioritized, practical checks and tools below to pinpoint the stuck syscall or code path. Targeted debugging checklist (step-by-step) 1. Confirm single-thread vs multi-thread behaviour ◦ Try running with a single Tokio worker thread (set RUST_LOG and tokio builder or env var if you use one) or force single-threaded. If it behaves differently, it points to a runtime/threading interaction. 1. Capture a snapshot of what threads are doing ◦ While the process is hung, run: ◦ ps aux | grep <pid> (confirm PID) ◦ sudo cat /proc/<pid>/stack (shows kernel stack for each thread) — helpful for blocking syscalls. ◦ sudo ls -l /proc/<pid>/fd (see open sockets / fds) ◦ Use the above to see if threads are waiting in the kernel (e.g., read, connect, futex). 1. Use strace to see blocking syscalls ◦ Run the process under strace or attach to a running process with sudo strace -ff -p <pid> -o /tmp/trace ◦ Look for syscalls such as connect(), accept(), recvfrom(), recvmsg(), poll/epoll_wait, open, or DNS-related calls that are stuck. ◦ If the process is already hung, the last syscall recorded will indicate where it’s blocked (for example, blocking in connect or in a TLS/crypto syscall invoked by native libraries). 1. Use gdb or lldb to get native backtraces ◦ Attach gdb to the running pid: sudo gdb -p <pid> ◦ Run
thread apply all bt full
to get stack traces for all threads. Native frames inside libwebrtc, rustls, or libc will show where it’s stuck. ◦ If you see frames in OpenSSL/rustls/libwebrtc, that’s strong evidence the native layer is blocking. 1. Check dynamic libraries and symbol versions ◦ ldd target/debug/your_binary or ldd /proc/<pid>/exe — verify which TLS/crypto libs are used (system OpenSSL vs vendored). ◦ If webrtc or rustls forks are built against system libs, ABI or platform-specific mismatches can cause strange hangs. 1. Reproduce with minimal code path ◦ Create a tiny sample that only establishes the WSS connection used by SignalClient (bypass libwebrtc if possible) — e.g., use async-tungstenite or tokio-tls to open a WSS to your server using the same URL. If the minimal WSS client also hangs, the problem is the TLS/WebSocket stack or network path rather than LiveKit-specific code. 1. Exercise timeouts at different layers ◦ Replace livekit-runtime timeout with your own explicit task and use
tokio::spawn_blocking
for suspect blocking calls. If the blocking call runs inside
spawn_blocking
and still prevents the OS watchdog from waking, that’s an important clue (it shouldn’t; spawn_blocking moves work to a blocking pool). 1. Try a pure-std watchdog that examines process status ◦ Instead of simply sleeping, have the watchdog thread periodically read /proc/<pid>/stack or send SIGQUIT to dump backtraces if still hung. A SIGQUIT (kill -3) will cause Rust programs to print backtraces when enabled. 1. Enable extended logging and TLS debug ◦ Turn on debug logging for TLS or the WebSocket crate used by the signal client (if configurable). If rustls is used, enable its debug logging if available. 1. Test a different runtime / environment ◦ Try running the same binary on an x86 VM on the same network to separate network from platform issues. ◦ Try building with different feature flags: disable libwebrtc or use a different TLS backend if possible. 1. Try a polling workaround (what you started) ◦ Your polling approach (is_finished() + sleep(100ms)) is a good test. If that also hangs, the issue is not Tokio’s timer wheel. If the polling loop still advances, it indicates timers were the culprit. 1. Snapshot core dump for offline analysis ◦ If you can’t attach live, enable core dumps (ulimit -c unlimited) and collect core when it hangs. Analyze with gdb to inspect threads and native stacks. Immediate practical checks I recommend you run first • While the process is stuck, run: sudo cat /proc/<pid>/stack and sudo ls -l /proc/<pid>/fd • Attach strace: sudo strace -ff -p <pid> -o /tmp/strace • Attach gdb and run thread backtraces: sudo gdb -p <pid> then thread apply all bt • Try your minimal WSS-only reproducer on the same device What to look for in traces/backtraces • Threads blocked in connect(), poll(), epoll_wait() are network I/O related. • Threads blocked deep inside libwebrtc, OpenSSL, or rustls are TLS/crypto related. • Futex or memory allocator deadlocks may show up as futex waits across threads. • Frames showing syscall getrandom or RNG could indicate blocking during entropy gathering (rare on modern kernels but possible in early-boot or misconfigured systems). If you capture traces or backtraces • Share the thread backtraces (native frames) or the last strace syscalls. The key indicators are which syscall is blocked and which library function is on the stack (libwebrtc, rustls, openssl, libc). Other notes and hypotheses • The fact that std::thread sleep also didn’t wake suggests the process may be blocked in kernel context (syscall) or a low-level deadlock rather than just an async timer waker problem. • libwebrtc contains native C++ code; if a blocking call happens there (for example certificate verification that calls into system libraries), it could stop progress and explain the symptoms. • If your system uses some hardware crypto or odd kernel modules on ARM64, those can behave differently than x86 and cause platform-specific hangs. If you want, I can: • Walk through reading a sample strace or gdb backtrace you post and point to which library/function looks suspicious. • Suggest exact strace/gdb commands to run tailored to your environment. • Help craft a minimal WSS-only repro and a binary you can run to isolate signaling from full SDK/media stack. Would you like the exact commands to attach gdb/strace and the most useful backtrace snippets to paste here? If so, tell me the PID when it’s hung (or paste a short strace/gdb output) and I’ll point out the relevant frames and next actions. Sources: Rust SDKs | livekit/livekit_composite | DeepWiki | Handling asynchronous metadata updates in LiveKit | LiveKit | Firewall Tips for LiveKit Connectivity | LiveKit