Thanks — you’re asking for a detailed, end‑to‑end description of the LiveKit pipeline. I’ll summarize the full flow and key components so you can see how media, signaling, AI processing, data, and integrations move through the system. Overview — what I’ll cover • Client connection and SDKs • Signaling (control plane) • Media transport and SFU (data plane) • Data tracks and non-media payloads • AI voice‑agent pipeline (VAD → STT → LLM → TTS) • Egress/Ingress/SIP integrations • Security (E2EE vs. secure trunking) • Deployment, scaling, and latency considerations 1) Client connection and SDKs • Clients use LiveKit SDKs (web, iOS, Android, React Native, Flutter, Unity, etc.) to open a room session and manage Participants and Tracks. • SDKs handle WebRTC primitives: ICE, DTLS, SRTP, and local media capture/publishing. 2) Signaling (control plane) • Signaling runs over WebSocket with protobuf messages (join, addTrack, updateSubscription, streamState updates). • Signaling coordinates room lifecycle, participant presence, track publication, and subscription management. 3) Media transport and SFU (data plane) • Media is sent as RTP over secure DTLS/SRTP channels to LiveKit’s SFU. • SFU performs selective forwarding (no full-bridge transcoding by default): it receives RTP from publishers and forwards appropriate RTP streams to subscribers. • Bandwidth adaptation uses simulcast/SVC and bandwidth estimation to choose quality layers per subscriber. • Clients decode and render the forwarded streams. 4) Data tracks (non-media) • DataTracks publish arbitrary binary or control frames over WebRTC data channels. • Frames use a compact header/packet format and are forwarded by the SFU similar to media (unordered/lossy semantics). • Data tracks support end‑to‑end encryption when configured; SFU forwards ciphertext only. 5) AI voice‑agent pipeline (low‑latency streaming) • Designed as a streaming, overlapping pipeline to minimize perceptible latency: a. Voice Activity Detection (VAD) — detects speech segments and turn boundaries (adds small latency). b. Speech-to-Text (STT) — streaming transcription produces partial results quickly. c. LLM / Dialog Manager — consumes transcript + chat context, can call tools, and streams token output as available. d. Text-to-Speech (TTS) — streaming TTS synthesizes audio chunks progressively. e. Audio delivery — synthesized audio is injected back into the room via WebRTC (or delivered via SIP). • Streaming overlap (VAD/STT/LLM/TTS concurrently) reduces total latency compared to naive sequential processing. 6) Integrations: Egress, Ingress, SIP • Ingress: bring external RTMP/WHIP streams into rooms. • Egress: record sessions or stream rooms out to RTMP endpoints (YouTube/Twitch) or recordings. • SIP: connect telephone calls into rooms; SIP calls can be bridged into WebRTC participants. 7) Security: E2EE vs. secure trunking • E2EE: client-side encryption of media/data — SFU forwards encrypted packets and never sees plaintext. • Secure trunking/SIP: when cloud processing (noise suppression, transcoding) is required, audio may be decrypted in the cloud for processing and re‑encrypted for WebRTC delivery — not E2EE. 8) Deployment, scaling, and latency • LiveKit Cloud uses regional edge nodes; clients connect to nearest edge to reduce network latency. • Edge nodes and private backbone reduce inter-region hops; SFU-based routing scales better than P2P for many participants. • Streaming AI pipeline and simulcast/SVC optimize perceived latency — typical streaming pipeline latencies are noticeably lower than naive sequential approaches. If you’d like, I can: • Draw this as a sequence diagram (client → SFU → AI services → SFU → client). • Provide sample SDK calls for join/publish/subscribe and for injecting synthesized audio. • Walk through a concrete example (web client + streaming STT + LLM + TTS) with latency numbers. Which of those would you like next? Sources: Sequential Pipeline Architecture for Voice Agents | LiveKit | Sequential Pipeline Architecture for Voice Agents | LiveKit | Introducing data tracks: realtime streaming beyond audio and video | LiveKi... | SIP Telephony and Encryption | Decentraland's Catalyst: using WebRTC to interact in the metaverse | LiveKi..., with interactive elements