I have custom ultravox model api hosted elsewere as vllm interface - how to integrate that with livekit as realtime

How is this going to handle multi-turn conversation? Do we need to run ASR in parallel? What if I want Websocket connection, no Auth, interim streaming as well for multi-turn conversation handling

Thanks — great details. I’ll cover multi‑turn handling, whether you need parallel ASR, and a minimal WebSocket + interim streaming approach (no auth). I’ll assume Python; tell me if you prefer Node. Short answers • Multi‑turn: keep and send conversation history (chat messages, speaker turns, system prompts) with each user turn so the model has context. Also manage turn boundaries (endpointing) so you know when to append final user turns. • Parallel ASR: not required. Your realtime LLM already does audio→text with interim results, so it replaces ASR. You only need a TTS for spoken replies. • WebSocket + no auth + interim streaming: supported — connect LiveKit audio frames to your WS, stream audio frames, parse interim text events, yield incremental ChatChunks to LiveKit, and finalize turns when you receive final transcripts or an end‑of‑utterance signal. What to implement (overview) 1. Audio framing: convert LiveKit audio frames to the PCM format your model expects (sample rate, mono, PCM16). Send frames as binary messages over WS. 2. Interim vs final: ◦ On interim transcript messages: yield incremental ChatChunks so LiveKit can start TTS or show partial text. ◦ On final transcript: mark the chunk final (so you append to conversation history). 1. Conversation state: ◦ Keep an ordered list of messages: system prompts, user messages (final only), and assistant messages (model outputs). ◦ For each new user utterance, include relevant recent history (sliding window) when continuing the session or when calling model APIs that require full context. 1. Turn management: ◦ Use model final transcript or explicit endpointing event from the WS to signal turn end. ◦ Optionally implement voice activity detection or silence timeout as a fallback. 1. TTS: when the model emits assistant text (interim or final, depending on UX), send to Cartesia TTS and pipe audio back to the room. 2. Backpressure & latency: ◦ Send audio in small frames (e.g., 20–40 ms) and yield interim text as soon as available. ◦ Buffering: avoid aggregating large buffers; stream continuously. Minimal Python WebSocket integration sketch • This is a skeleton showing WS audio send, handling interim/final messages, and maintaining conversation state. Fill in actual audio framing and message parsing per your model’s protocol.

import asyncio
import websockets
from livekit.agents import AgentSession, llm

class RealtimeWSLLM:
    def __init__(self, ws_url):
        self.ws_url = ws_url
        self.history = []  # list of dicts: {"role":"user"/"assistant"/"system","text":...}

    async def _send_audio_frames(self, ws, audio_frame_iter):
        async for frame in audio_frame_iter:  # frames from LiveKit
            pcm_bytes = convert_frame_to_pcm16(frame)  # implement per your audio format
            await ws.send(pcm_bytes)

        # signal end of stream if your protocol expects it
        await ws.send(b'--end--')

    async def stream_audio_and_receive(self, audio_frame_iter, on_chunk):
        """
        audio_frame_iter: async iterable of audio frames from LiveKit
        on_chunk: callback async fn receiving dicts like {"text": "...", "interim": True/False}
        """
        async with websockets.connect(self.ws_url) as ws:
            sender = asyncio.create_task(self._send_audio_frames(ws, audio_frame_iter))

            async for raw_msg in ws:  # server sends interim/final JSON messages
                msg = parse_server_message(raw_msg)  # {"text": "...", "type":"interim"/"final"/"assistant"}
                if msg["type"] in ("interim", "partial"):
                    await on_chunk({"text": msg["text"], "interim": True})
                elif msg["type"] in ("final", "utterance_end"):
                    # commit final user turn to history
                    self.history.append({"role":"user", "text": msg["text"]})
                    # if model also returns assistant reply in same message:
                    if msg.get("assistant_text"):
                        assistant_text = msg["assistant_text"]
                        self.history.append({"role":"assistant", "text": assistant_text})
                        await on_chunk({"text": assistant_text, "interim": False, "from_assistant": True})
                    else:
                        await on_chunk({"text": msg["text"], "interim": False})

            await sender

How conversation turns map to LiveKit nodes • Interim transcripts → yield llm.ChatChunk with intermediate content so UI/TTS can react. • Final transcript → append to history and mark chunk final so future prompts include it. • Assistant outputs → append to history and pass to TTS for playback. Practical tips • Keep a sliding window of recent messages to limit context size (e.g., last N exchanges). • If your LLM can accept prior text context over the same WS, send history when starting a new turn; otherwise, store it server-side and prepend to prompts. • For latency: synthesize TTS from interim assistant text if you want faster replies, but be prepared to cancel/replace audio when the final assistant text differs. • Implement reconnection logic and idempotence for the WS. Next, I can: • Provide a runnable Python example that integrates with LiveKit audio frames (if you share the exact frame format LiveKit provides), or • Convert the sketch to Node if you prefer. Which do you want next — a full runnable Python example tied to LiveKit audio frame APIs, or a Node version? , with interactive elements

I’m designing a custom LiveKit realtime model integration for a self-hosted Ultravox/vLLM audio endpoint over WebSocket. The endpoint receives audio frames and streams back: • user transcript interim/final events • assistant text delta/final events My question is about conversation history ownership in LiveKit Agents. In this architecture, should LiveKit’s

AgentSession

/ chat context be the source of truth for multi-turn history and actively feed final user + assistant messages back into the realtime model endpoint each turn? Or, if the model endpoint keeps its own session state inside the persistent WebSocket connection, is it enough for LiveKit to only mirror final user/assistant text into its own chat context / transcript for tools, handoffs, UI, and

transcript.json

, while the model’s next-turn context remains server-side? More specifically: 1. In LiveKit Agents, what internal history/state is expected to be maintained for custom realtime model integrations? 2. Does

AgentSession

automatically use its chat context to condition the next realtime model turn, or does the custom realtime model adapter need to explicitly send that history to the model endpoint? 3. For a WebSocket-based realtime model that is already sessionful, is there any downside to treating LiveKit history as a mirror rather than the model’s actual context source? 4. What is the recommended pattern for committing: • final user transcripts, • assistant final text, • interim user transcripts, • assistant deltas, into LiveKit history/transcription events? 5. If the model endpoint is stateless instead, what LiveKit hook/node should be responsible for injecting prior messages back into the model request? I’m trying to choose between two designs: Design A — sessionful model: LiveKit mirrors final transcripts/responses for transcript/tools/UI, but the model remembers context inside the WebSocket session. Design B — LiveKit-owned history: LiveKit is the source of truth and sends prior final messages back to the model each turn. Which design fits LiveKit Agents best, and what should the custom adapter implement?

Thanks — great questions. I’ll answer each point directly and then give a clear recommendation and required responsibilities for your custom WebSocket realtime adapter (no auth, interim streaming, model returns user interim/final + assistant delta/final). 1) What LiveKit expects you to maintain internally • LiveKit Agents expects a ChatContext (

chat_ctx

) to be the canonical conversation record inside AgentSession: ordered messages with roles (

system

,

user

,

assistant

), function calls/outputs, handoff markers, etc. • The framework and many features (tooling, handoffs, transcript.json, context copy/truncate/merge) rely on that

chat_ctx

being accurate and up-to-date. 2) Does AgentSession automatically condition the model on chat history? • No. AgentSession does not magically push chat context into your realtime model. Your custom realtime adapter must explicitly send any history the model needs for conditioning on the next turn (unless the model itself is sessionful and already holds context server‑side). 3) If the model is sessionful, any downsides to mirroring only? • Mirroring (Design A: model holds canonical state; LiveKit mirrors final texts for UI/tools) works but has tradeoffs: ◦ Pros: lower per‑turn payloads and latency; simpler WS protocol. ◦ Cons: breaks LiveKit-native features that expect LiveKit-owned context (tooling/function calls, agent handoff, context copy/truncate/inspect). Recovery/replication is harder if the model connection resets. Harder to run pipeline nodes that need full history. • If you choose Design A, you must ensure robust reconciliation, resync on reconnect, and extra plumbing so LiveKit features (tools, handoff) can access required context or trigger model-side operations. 4) Recommended event-commit pattern (what to write into LiveKit history) • Final user transcripts: commit immediately to

chat_ctx

as a

user

message when you receive the model’s final user-turn event (this is the authoritative user turn for tools, transcripts, and handoffs). • Assistant final text: commit immediately to

chat_ctx

as an

assistant

message when the model emits final assistant text. • Interim user transcripts: publish as transient transcription events (e.g.,

lk.transcription

stream) and optionally emit interim

llm.ChatChunk

so UI/UX can show partial text, but do NOT commit interim partials as final

user

messages. • Assistant deltas (interim assistant text): stream as incremental `ChatChunk`/transcription so the UI/TTS can react; commit only the assistant final when the model emits a final assistant event. • For tools/function-calls: only insert function_call and function_call_output into

chat_ctx

when the call is finalized and the output is available. 5) If the model is stateless, where to inject prior messages? • Use LiveKit pipeline hooks/nodes that run when a user turn completes (e.g.,

on_user_turn_completed

/

llm_node

or equivalent node). Those nodes receive the

ChatContext

and the new final user message; your adapter should serialize and send the relevant slice of

chat_ctx

to your stateless endpoint when initiating the next turn. • Implement context truncation/selection here (sliding window, instruction preservation) before sending to the model. Recommended pattern (what fits LiveKit best) • Prefer Design B (LiveKit-owned history) for best compatibility: ◦ Keep LiveKit

chat_ctx

as the source of truth. ◦ On each final user/assistant event, commit into

chat_ctx

. ◦ For stateless endpoints, the adapter pulls

chat_ctx

and sends prior messages when starting a model turn. ◦ For sessionful endpoints that prefer server-side context, still mirror finals into

chat_ctx

, but also send a copy of the same finals to the model on session start (or rely on the model maintaining consistency); implement resync on reconnect. • Why B: it preserves tool/function integration, handoffs, pipeline operations (copy/truncate), observability, and easier recovery. Adapter responsibilities (concrete checklist) • Maintain WS connection, send audio frames, parse interim/final messages. • Emit interim events to LiveKit transcription/ChatChunk streams (for UI/TTS). • Commit final user messages and final assistant messages to

chat_ctx

immediately. • If your model is stateless: before starting a new assistant generation, send the selected

chat_ctx

history slice to the model. • If your model is sessionful: on connection or reconnection, send a resync of final messages from

chat_ctx

to the model or provide a reconciliation mechanism. • Implement robust reconnection, idempotency, and sequence numbers so LiveKit ↔️ model state stays consistent. • Support cancellation of interim TTS outputs if assistant text changes between interim and final. Practical tips • Use a sliding-window policy (N last turns or X tokens) when sending history to limit size. • Keep interim streaming separate from committed history: stream for UX, commit only finals. • On reconnect, prefer to rehydrate model state from LiveKit

chat_ctx

(even if your model had state) to avoid divergence. • For handoffs/tooling, always rely on

chat_ctx

contents — do not assume model-side-only state will be visible to LiveKit features. Conclusion (short) • LiveKit works best when it owns the canonical history (Design B). If you must run a sessionful model (Design A), mirror finals into LiveKit and implement explicit resync/reconciliation and adapters so LiveKit features continue to function reliably. Your adapter must explicitly handle context injection (for stateless) or resync (for sessionful), commit finals into

chat_ctx

, stream interim events for UX, and implement reconnection/idempotency. If you want, I can: • Provide a minimal adapter template (WebSocket Python) that demonstrates interim handling, final commits to

chat_ctx

, and a resync flow for sessionful models. • Or show the

on_user_turn_completed

node example that injects

chat_ctx

into a stateless request. Which would you prefer? Sources: Chat context | LiveKit Documentation | Realtime models overview | LiveKit Documentation | Text and transcriptions | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation, with interactive elements

I’m trying to verify whether we need a custom gateway/service between LiveKit Agents and a self-hosted vLLM audio model, or whether this can be done natively with existing LiveKit/vLLM capabilities. Target architecture: Browser/caller → LiveKit Room → LiveKit AgentSession → self-hosted realtime audio model served via vLLM → streamed assistant text → LiveKit TTS node → LiveKit Room. The model should consume live microphone audio and emit: • user transcript interim events • user transcript final events • assistant text delta events • assistant final text events Questions: 1. Does LiveKit Agents already support connecting a custom realtime audio-input/text-output model directly, without a separate STT node? 2. Is the correct integration point a custom

RealtimeModel

/

RealtimeSession

, a custom

llm_node

, or something else? 3. Can LiveKit stream `rtc.AudioFrame`s directly into a custom realtime model session? 4. Can the custom realtime model emit user interim/final transcription events into LiveKit’s normal transcription lifecycle? 5. Can assistant text deltas from the custom realtime model be streamed directly into LiveKit’s existing TTS node? 6. If the model endpoint is already WebSocket-based and sessionful, does LiveKit still require an adapter layer, or can the custom realtime model connect to it directly? 7. Does LiveKit provide built-in support for session updates/history sync into realtime models, or must the adapter implement that explicitly? 8. Does LiveKit automatically commit final user transcripts and assistant final responses to

chat_ctx

, or must the custom realtime model/session do that? 9. Are interruption, cancellation, and stale-event handling handled by LiveKit for custom realtime models, or must the adapter implement them? 10. Is there any official example of a custom realtime model that consumes audio directly and outputs text to the TTS pipeline? I’m specifically trying to decide whether we need a separate protocol-normalizing service between LiveKit and the model server, or whether a LiveKit custom realtime adapter can talk directly to the model endpoint.

Thanks — concise answers for each question, plus a final recommendation. 01) Native support without a separate STT node? • Yes. LiveKit Agents supports audio‑in → text‑out realtime models via a custom realtime plugin; you can replace STT+LLM with a single realtime model adapter so no separate STT node is required. 02) Correct integration point? • Implement a custom RealtimeModel + RealtimeSession. Don’t use

llm_node

for full audio→text realtime integration;

RealtimeSession

manages the WS, audio streaming, and event mapping. 03) Can LiveKit stream `rtc.AudioFrame`s directly to the model? • Yes. A RealtimeSession can read LiveKit audio tracks (rtc.AudioFrame) and forward frames to your WebSocket/streaming endpoint. 04) Can the model emit user interim/final transcription into LiveKit’s lifecycle? • Yes. Your session should emit the appropriate events (e.g.,

user_transcription

interim and

user_transcription_final

) so LiveKit and clients receive transcriptions and LiveKit can publish them. 05) Can assistant text deltas be streamed into LiveKit’s TTS node? • Yes. Emit assistant text delta (interim) and assistant final events from the session; LiveKit can route text to a configured TTS node for incremental/final speech synthesis (half‑duplex flow). 06) If the model is WebSocket/sessionful, is an adapter still required? • You still need a custom RealtimeSession adapter inside LiveKit Agents to speak LiveKit’s event/model contract and manage audio/frame conversion, event emission, reconnection, and mapping to chat_ctx. That adapter is the gateway — a separate external service is not necessary unless you want centralized translation, load balancing, or shared connections. 07) Does LiveKit provide built-in session/history sync? • No. LiveKit does not automatically sync history into custom realtime models. The adapter must explicitly implement history loading, context injection, or resync logic (for stateless or sessionful models). 08) Does LiveKit auto-commit final transcripts/responses to

chat_ctx

? • Only if your session emits the final events. When the session emits final user/assistant events (the well‑known final event types), LiveKit will commit those into

chat_ctx

. The adapter must emit them. 09) Are interruption, cancellation, and stale-event handling handled automatically? • Partially. LiveKit provides TTS cancellation, session interrupt signals, and infrastructure hooks, but the model-level interruption/cancel/stale-event handling must be implemented by your adapter (e.g., send abort/generation stop to model, cancel TTS audio, reconcile partials). 10) Is there an official example of a custom audio-in/text-out realtime plugin? • No single generic vLLM example, but there are official realtime plugin examples (OpenAI Realtime, Grok, PersonaPlex) that demonstrate the exact pattern (WebSocket session, audio frame forwarding, transcription events, assistant text streaming). Use those as templates. Practical recommendations / checklist • Implement a custom RealtimeModel + RealtimeSession inside your agent package. That is the adapter/gateway — no separate external service required unless you need centralized concerns. • Responsibilities for the adapter: ◦ Open/maintain the WS to your vLLM endpoint and stream

rtc.AudioFrame

-> required PCM bytes. ◦ Parse interim/final user transcription and emit

user_transcription

/

user_transcription_final

. ◦ Parse assistant deltas/final text and emit incremental LLM output and

llm_response_final

(or equivalent) so TTS can synthesize. ◦ Commit final user and assistant messages to

chat_ctx

when final events arrive. ◦ Implement optional history sync/resync (on connect/reconnect) if your model is sessionful, or send selected

chat_ctx

history per turn if stateless. ◦ Handle interruptions/cancels: accept LiveKit interrupt signals and translate them to model aborts, cancel TTS, and manage stale intermediate outputs. ◦ Implement context truncation/selection (sliding window) before sending history if the model is stateless or when rehydrating session state. ◦ Reconnection and idempotency: resync on WS reconnect to avoid divergence. Recommended design choice • Prefer LiveKit-owned history (LiveKit commits finals into

chat_ctx

) and have the adapter explicitly send context to the model when needed. This preserves tool/function integration, handoffs, and recovery. If you choose a sessionful model as the true source-of-truth, ensure your adapter provides robust resyncing and exposes the model state (or mirrors it) to LiveKit for tooling and handoffs. If you want, I can: • Produce a minimal runnable Python RealtimeSession template (WebSocket, no auth, interim + final parsing, commits to chat_ctx, resync flow) tuned to your exact WS message schema and audio frame format. Which message schema and pcm/frame format does your model expect? Sources: OpenAI Realtime API plugin guide | LiveKit Documentation | Realtime models overview | LiveKit Documentation | NVIDIA PersonaPlex plugin guide | LiveKit Documentation | Agent session | LiveKit Documentation | Introduction | LiveKit Documentation, with interactive elements