Thanks — this is a great, thorough set of questions. I can help verify each assumption, but the LiveKit docs do not fully confirm every low-level behavior for a hybrid adapter (external ASR websocket + separate chat completions + TTS). To give accurate, actionable guidance I need one quick clarification first: When you say the adapter will run as a LiveKit "RealtimeModel / RealtimeSession" extension, do you plan the adapter to: • fully implement LiveKit's realtime model interface (i.e., appear to LiveKit as a single model provider handling audio-in/transcription/generation events), OR • run as an external service that bridges LiveKit audio -> external ASR/LLM/TTS and communicates with LiveKit via pipeline node APIs (transcription_node / llm_node / tts node) rather than the built-in realtime model plugin interface? Tell me which of the two you intend; I’ll then answer all 20 questions precisely and point to the exact hooks/events you should use and where LiveKit behavior is unspecified so you can add safe, recommended handling. Sources: Realtime models overview | LiveKit Documentation | LiveKit turn detector plugin | LiveKit Documentation | Turns overview | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation, with interactive elements

Thanks — this is a great, concrete brief. I’ll design an in-process LiveKit-side adapter from first principles that meets your goals and works around the vLLM constraints. I’ll state where LiveKit docs are silent and give safe, recommended behaviors. Before I deliver the full design, quick confirmation: do you want the adapter written to run inside the LiveKit Agents process (in-process plugin) with direct access to AgentSession/RealtimeSession APIs, or should it run as a separate service that the LiveKit plugin calls (out-of-process gateway)? I’ll default to in-process unless you prefer otherwise. If in-process, I’ll proceed and cover each requested item (01–19) in order, including: • chosen extension point and rationale • turn-detection ownership and exact signals to use • audio ingestion and conversion pipeline (formats, sampling, buffering, commit timing) • mapping of vLLM transcription events into LiveKit UI vs canonical chat_ctx • where and how to call chat/completions and stream into TTS • capability flags to set • IDs/state to maintain for dedupe and stale-event protection • interruption, reconnect, history selection, tests, and APIs to avoid Confirm in-process and I’ll produce the full design now. , with interactive elements

Thanks — I’ll assume common sensible defaults and produce the full in‑process adapter design now. If any default is wrong, tell me which and I’ll adjust. Assumptions I’ll use: • Runtime: Node/TypeScript (common for LiveKit plugins). • Adapter can use in‑process AgentSession / RealtimeSession public plugin interfaces. • rtc.AudioFrame arrives as 48 kHz float32 stereo frames (common); vLLM requires 16 kHz PCM16 mono base64. • LiveKit VAD is available and emits turn-end events; adapter can also rely on vLLM

transcription.done

if needed. • vLLM realtime auth via API key; chat completions via HTTP(S) streaming. • Existing TTS node accepts incremental text chunks and supports cancelling playback. • chat_ctx persisted in LiveKit agent memory and exposed to admin APIs; persistence across restarts is out of scope (can be added externally). • Goal: single in‑process RealtimeModel + RealtimeSession adapter bridging audio→vLLM ASR → /chat/completions → LiveKit TTS, with canonical chat_ctx commits only for final user/assistant turns. Design (concise, ordered): 1) Extension point • Use custom RealtimeModel + RealtimeSession in‑process. Rationale: need direct access to rtc.AudioFrame stream, session lifecycle hooks, emission of transcription and generation events, and tight synchronous control for commit/turn semantics. 2) Turn-detection ownership • Primary owner: LiveKit VAD for user turn boundary detection. • Secondary confirmation: send vLLM

input_audio_buffer.commit

when LiveKit VAD signals end-of-turn. • Treat vLLM

transcription.done

as authoritative for ASR finalization only if it arrives after commit; do not rely solely on it for boundary detection because vLLM has no turn IDs. 3) Audio ingestion • RealtimeSession should receive pushed `rtc.AudioFrame`s from LiveKit (adapter registers a frame handler on session/track subscription). Implement per-participant audio streams keyed by participant ID. 4) Audio conversion pipeline • Convert per-frame: ◦ Deinterleave stereo → pick dominant speaker channel or mix to mono. ◦ Resample from 48 kHz float32 → 16 kHz. ◦ Convert float → PCM16 little-endian. ◦ Base64-encode PCM16 blocks sized to vLLM buffer targets (e.g., 100–200 ms). • Provide unit for conversions in adapter (do not depend on LiveKit utilities: assume none provided). 5) When to send input_audio_buffer.append / commit • Append continuously as audio frames arrive (batched into 100–200 ms chunks). • On LiveKit VAD end-of-turn: call

input_audio_buffer.commit

immediately. • If VAD absent or unreliable: use trailing silence timeout (e.g., 600 ms) then commit. 6) Exposing transcription.delta to UI without polluting chat_ctx • Emit LiveKit transcription events with a flag

interim=true

(or use

transcription.delta

event type). • Map these to UI-only channels (transcription UI stream) and do NOT append to chat_ctx. • Include metadata:

asr_session_id

,

asr_chunk_id

,

confidence

,

timestamp

. • Keep interim transcripts ephemeral in adapter state for UI replay; drop on final commit. 7) Converting transcription.done → final user turn • On

transcription.done

(matching latest

asr_commit_id

) and after VAD commit: ◦ Create a canonical user message object: { role: "user", text: final_transcript, asr_commit_id, turn_id }. ◦ Mark it final and emit an event

transcription.final

for UI. ◦ Do NOT auto-add to chat_ctx yet (see next). 8) Who commits final user message into chat_ctx • Adapter should explicitly append the final user message into chat_ctx via LiveKit API (call it

chat_ctx.append_message()

or equivalent). Rationale: LiveKit will not infer/merge from transcription events because vLLM lacks IDs; explicit append avoids ambiguity and gives the adapter control to dedupe. 9) Where/how to call /v1/chat/completions • After appending final user message to chat_ctx (ensures chat_ctx canonical ordering), adapter should call chat/completions: ◦ Build request with selected chat history (chat_ctx windowing policy) + new final user message ID reference. ◦ Open HTTP streaming connection; record

generation_id

. ◦ Emit generation start event into LiveKit with

generation_id

and linked

turn_id

. 10) Streaming assistant deltas into TTS pipeline • As streaming HTTP yields deltas: ◦ Publish each delta to LiveKit llm/generation stream as

llm.ChatChunk

or

MessageGeneration.text_stream

equivalent (incremental text, generation_id). ◦ Forward the same deltas to TTS node via its incremental API (e.g.,

tts.append_text_chunk(generation_id, chunk_text)

). • If TTS requires final text only, buffer deltas and flush final on completion. 11) Who commits final assistant message into chat_ctx • Adapter should append the final assistant message into chat_ctx only after the chat/completions stream completes successfully. • On completion, create assistant message object { role: "assistant", text: full_text, generation_id, turn_id } and append to chat_ctx. 12) Avoiding duplicate messages • Use immutable IDs: ◦ turn_id (per user turn) ◦ asr_commit_id (per vLLM commit) ◦ generation_id (per chat completion) • chat_ctx append operations must include the ID; append endpoint should be idempotent (adapter should check whether message with same ID exists before appending). • Maintain local index mapping IDs → appended flag; skip appends if already present. 13) Capability flags • audio_output=False (adapter does not synthesize audio itself). • user_transcription=True (adapter provides ASR transcripts). • turn_detection=False? No — set turn_detection=False only if LiveKit requires model to do turn detection. Since LiveKit VAD owns endpointing in our design, set

turn_detection=False

to indicate model does not perform turn detection. But the adapter still emits final user turns (it appends final transcripts) — this is fine:

turn_detection=False

means model won't claim to own live endpointing, not that it can't emit final messages. • mutable_chat_context=True (adapter will read and append to chat_ctx). 14) Internal IDs/state • Maintain: ◦ session_id (LiveKit AgentSession id) ◦ participant_id ◦ turn_id (UUID per VAD-detected user turn) ◦ asr_commit_id (UUID per vLLM commit) ◦ generation_id (UUID per chat/completions call) ◦ appended_user_message_ids, appended_assistant_message_ids (sets for idempotency) ◦ pending_generations map: generation_id → cancellation token / HTTP stream handle ◦ last_interim_sequence per participant for ordering 15) Interruption behavior • On user interrupt during assistant generation/TTS: ◦ Cancel TTS playback immediately via TTS node cancel API. ◦ Cancel the /v1/chat/completions stream (if supported). vLLM chat does support cancellation via closing connection; adapter should close HTTP stream and mark generation cancelled. ◦ Mark generation as cancelled and avoid appending partial assistant text to chat_ctx. ◦ Ignore late assistant deltas by checking generation_id and

cancelled

flag—do not forward them to TTS or append. • If chat completion cannot be cancelled server-side, close client-side stream and discard incoming bytes; TTS must be cancelled locally. 16) Reconnect to vLLM / resiliency • Maintain short-lived vLLM realtime connections per participant session. • On ASR WS reconnect: ◦ Recreate a fresh vLLM session with new

asr_session_id

. ◦ Resume streaming new audio; do NOT resend old audio; vLLM has no memory anyway. ◦ If there was an in-progress asr_commit that never returned

transcription.done

, mark it failed and notify UI. • Keep retry/backoff and surface reconnect status to UI. 17) Conversation history selection • Adapter controls windowing policy: ◦ Keep last N messages by tokens or time, prefer assistant + final user messages only. ◦ When calling chat/completions, fetch canonical chat_ctx (adapter-read) and select messages per policy. ◦ Include message IDs so chat/completions responses can be linked back to chat_ctx entries on append. 18) Tests to write • Unit: ◦ Audio conversion: stereo→mono, resample, float→PCM16, base64 correctness. ◦ ID handling: idempotent append behavior. ◦ Interim vs final event routing. • Integration: ◦ Full flow: audio -> vLLM WS (mock) emits deltas/done -> adapter appends user -> calls chat/completions (mock) streams assistant deltas -> TTS receives chunks -> final append. ◦ Interrupt test: while streaming assistant, send user audio -> adapter cancels generation and TTS. ◦ Reconnect test: drop ASR WS mid-turn -> reconnect -> new asr_session. • E2E: ◦ UI sees interim transcripts, final commits in chat history, no duplicates. 19) LiveKit SDK limitations / APIs to avoid • Avoid relying on undocumented/private methods or internal state mutation APIs. • Do not assume LiveKit will auto-commit transcription events to chat_ctx. • Do not rely on LiveKit exposing low-level audio format conversion utilities — implement adapter-side conversions. • Avoid assuming server-side cancellation behavior for external chat endpoints. Operational notes , with interactive elements

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!