i have a question about the livekit infrastructure and If i can have a connection using a websocket myself to realtime models myself?

Thanks for reaching out, Joshua. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

im running a telephony setup and i want to be able to connect my websocket and connect this websocket to aanother custom model for realtime streaming can I do that??

Thanks — that helps. Quick answer: you cannot have a client (phone/browser) directly stream media to LiveKit or LiveKit’s models over a raw WebSocket. However, you can accomplish what you want by running a server-side agent that bridges between telephony, LiveKit, and your custom realtime model. What you can do: • Run an agent service (Node/Python) that joins a LiveKit room as a participant using WebRTC (so media routing, mixing, and client compatibility are handled). • Have that agent open a WebSocket to your custom realtime model provider and forward audio in real time (and forward model responses back into the LiveKit room). • For telephony, use LiveKit’s SIP/telephony integration or ingest telephony audio into the agent so the call media is bridged into the LiveKit room and then to your model. • Alternatively, use Track Egress to forward raw PCM audio from LiveKit to your server over WebSocket for one-way streaming (useful for transcription or external processing), but Track Egress is outbound only — you’d still need an agent to send processed audio back into the room. If you’d like, tell me: • How telephony is connected now (SIP gateway, PSTN provider, SIP trunk, etc.). • Whether you need bidirectional audio between the call and the model (both directions). • Which realtime model/protocol you plan to use on your WebSocket. I can then outline a minimal architecture and example steps (or sample code) to implement the bridge. , with interactive elements

yeah bidirectional audio steaming yes

Thanks — responding to your confirmation that you need bidirectional audio streaming from telephony → model → call. Short answer: Yes — but not by streaming raw media over a client WebSocket. You should run a server-side agent (Node/Python) that bridges telephony, LiveKit, and your custom realtime model: • Agent joins a LiveKit room as a participant using WebRTC (handles media routing to/from callers). • Agent sends client audio to your custom model over a WebSocket (or whatever model protocol) and receives model audio back. • Agent forwards model audio into the LiveKit room so callers hear responses (true bidirectional streaming). • Optionally use Track Egress to stream raw PCM out to your server if you only need one-way export, but egress alone can’t return audio into the room — the agent is required for true two-way. If you’d like, I can outline: 1) A minimal architecture diagram and data flow. 2) Example steps or sample code (Node or Python) for: joining LiveKit, capturing audio, opening a WebSocket to your model, forwarding PCM/Opus frames, and injecting model audio back into the room. Quick questions to tailor the examples: • How is telephony connected today (SIP gateway, PSTN provider, WebRTC client)? • Which realtime model/protocol does your model expect (raw PCM frames, Opus, RTP over WebSocket, custom JSON framing)? • Do you need low-level audio format conversion (e.g., 16kHz PCM ↔️ model) or codec passthrough? Pick one or two answers and I’ll give a focused architecture and code snippets. , with interactive elements

yeah 16khz over a livekit number or something yeah and I want pstn not webrtc

Thanks — I’ll respond to your goal: bidirectional 16 kHz audio between PSTN (via a LiveKit number) and your custom realtime model over WebSocket. Recommended approach (summary) • Run a server-side agent (Node or Python) that acts as the bridge. • Use LiveKit Telephony/SIP (LiveKit number) to bring the PSTN call into a LiveKit room. • Agent joins that room as a participant via WebRTC, subscribes to the caller’s audio track, and publishes audio back into the room. • Agent opens a WebSocket to your custom realtime model. It: ◦ Sends caller audio to the model as 16 kHz PCM (perform resampling/format conversion if needed). ◦ Receives model audio (16 kHz PCM or your chosen codec) and injects it into the LiveKit room as an outgoing track. • This yields true bidirectional streaming: PSTN → LiveKit → Agent → Model and Model → Agent → LiveKit → PSTN. Key implementation points • Telephony: LiveKit handles SIP/PSTN ingress when you configure a LiveKit phone number or SIP gateway; the call appears in a LiveKit room as a participant’s audio track. • Media transport: LiveKit uses WebRTC for media. Your agent must use WebRTC to join the room (not a raw WebSocket for media). • Audio format: ◦ Convert incoming audio to 16 kHz PCM if the model requires it. ◦ Prefer raw PCM frames or Opus depending on your model’s protocol — but if your model needs 16 kHz PCM, resample and send PCM. • Codecs & latency: ◦ If your model accepts Opus, consider forwarding Opus frames to reduce CPU/resample costs. Otherwise decode to PCM and re-encode as needed. ◦ Keep buffer sizes small for low latency; use streaming WebSocket frames rather than large aggregated uploads. • Synchronization & RTP timing: ◦ If your model expects RTP/clock information, you’ll need to handle RTP timestamp mapping; otherwise use simple frame streaming with per-frame timestamps or sequence IDs. • Track egress vs agent: ◦ Track Egress can push raw audio to your server over WebSocket (one-way). For bidirectional audio you must run an agent to publish model audio back into the room. • Telephony quirks: ◦ PSTN latency and packetization differ from WebRTC; add jitter buffering and adaptive playout management in the agent. ◦ Ensure DTMF and call-control events are forwarded/handled if needed. Next steps I can provide • A minimal architecture diagram and sequence of events (telephony -> LiveKit -> agent -> model). • Sample code (Node or Python) showing: a. Agent joining LiveKit and subscribing/publishing audio tracks. b. Resampling/encoding pipeline for 16 kHz PCM. c. WebSocket framing for sending/receiving audio to your model. • Questions to tailor samples: ◦ Which language do you prefer for the agent: Node or Python? ◦ Does your model accept 16 kHz PCM, Opus, or RTP-over-WebSocket? Any framing/protocol details? Tell me preferred language and model protocol and I’ll return a focused architecture plus a short sample implementation. , with interactive elements