We have a custom audio-input LLM served through vLLM. The model can accept user audio through

/v1/chat/completions

, but the server does not expose

/v1/realtime

or any WebSocket realtime endpoint. We need to use this model in LiveKit Agents for a voice assistant. Requirements: • user speaks in LiveKit Room; • model receives the user’s audio, not only text; • assistant text streams back and is spoken through our existing TTS; • multi-turn conversation must work; • user transcripts should appear in LiveKit UI/transcript history; • final user and assistant messages should be stored correctly in `chat_ctx`; • avoid private LiveKit APIs if possible. Given that the model itself does not provide a realtime transcript stream, what LiveKit architecture would you recommend? Should we use: 1. standard STT + LLM + TTS pipeline, 2. custom

RealtimeModel

/

RealtimeSession

, 3. parallel ASR plus audio-input LLM, 4. or another pattern? Please explain the recommended design, who owns user transcription/history, how the current turn audio reaches the model, and how multi-turn context should be maintained.

Thanks for reaching out, Andrii. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

The key implementation question now is: in LiveKit Agents Python, what is the clean public way for a custom

llm_node

to access the current user turn’s raw audio buffer? We want the standard STT node to keep producing transcripts/UI/chat history, but the custom LLM call must send the original user audio to

/v1/chat/completions

. Questions: 1. Does

llm_node

receive or expose the finalized user-turn audio buffer through any public API? 2. If not, where should we tap/buffer audio frames using public LiveKit APIs? 3. Can we buffer audio in a custom STT wrapper while still using the STT provider normally? 4. Can we access the same audio frames from

AgentSession

or a pipeline hook without subscribing to room tracks manually? 5. How should we align the buffered audio turn with the final STT transcript / chat_ctx user message? 6. What public event should signal “this buffered audio belongs to this finalized user turn”? 7. Should the audio-buffering component live in: ◦ custom STT node/wrapper, ◦ custom LLM node, ◦ Agent/AgentSession hook, ◦ custom pipeline node, ◦ or a custom RealtimeSession? 8. Which private/internal APIs should we avoid for this? Please propose the cleanest Python LiveKit implementation pattern for “standard STT for transcript + custom LLM node that needs current-turn audio”.

Thanks — good questions. Below is a concise, public-API–only pattern for LiveKit Agents (Python) that meets your goal: keep standard STT for UI/transcripts while letting a custom

llm_node

send the current turn’s raw audio to your vLLM REST endpoint. Short answers to your numbered questions 1. No —

llm_node

does not automatically receive raw audio buffers via a public API. 2. Buffer audio in a custom STT wrapper/node (public STT node hook) that decorates the standard STT node. 3. Yes — you can buffer audio in a custom STT wrapper while delegating actual transcription to the underlying STT provider. 4. No — don’t rely on hidden AgentSession internals. Use the public STT node + Agent lifecycle hooks to share buffered audio with the rest of the session. 5. Align by using the STT turn-final signal (final transcript / END_OF_SPEECH / final transcript event) to mark which buffered audio belongs to that finalized user message, then attach both audio and final transcript to the same turn record. 6. Use the public STT “final transcript” or “end-of-speech” event (the standard final-transcript event your STT node yields) as the signal that “this buffered audio belongs to this finalized user turn.” 7. Best place for audio-buffering: primary logic in a custom STT node/wrapper, publish a turn-complete artifact via an Agent hook; LLM node reads that artifact. Do NOT use RealtimeSession for this when your model lacks WebSocket realtime support. 8. Avoid private/internal APIs like unpublished AgentSession internals, private track subscription hacks, or internal buffer variables. Rely on public node hooks,

rtc

audio frame handling APIs, and Agent lifecycle hooks. Clean implementation pattern (step-by-step) 1) Create a custom STT wrapper node • Implement a node that wraps the standard STT node. Public API signature remains the same so LiveKit still uses it for UI transcripts. • As frames arrive, append them to an in-memory buffer for the current turn (e.g., list of frames or a streaming buffer). Also forward frames to the wrapped STT processor so transcription proceeds normally and transcripts are published to the UI. 2) Detect turn boundaries via public STT events • Watch for the standard STT “final transcript” /

END_OF_SPEECH

event emitted by the underlying STT node. • When that final event is received: ◦ Snapshot the buffered audio for the turn. ◦ Capture the final transcript text emitted by STT. ◦ Clear the buffer for the next turn. 3) Emit a public turn-complete artifact • On final transcript, attach a turn object to the Agent instance (public field you manage), or better: call a documented Agent lifecycle hook (e.g., on_user_turn_completed / similar public hook) with a small object: ◦ { audio_bytes: wav/pcm bytes or frames, transcript: "final text", timestamp, turn_id } • This is your canonical signal proving "this audio belongs to this final transcript." 4) Convert frames to a transfer format • In the STT wrapper, convert buffered frames to the audio format your vLLM expects (wav/pcm/opus). Use public helper functions for combining frames and converting to WAV bytes before handing to the LLM node. 5) Make

llm_node

read the buffered audio on turn completion • In your custom

llm_node

, before calling vLLM, check whether the Agent has a pending turn-complete artifact (the one emitted in step 3). • If present, include the artifact.audio_bytes (and optionally metadata like sample rate or turn_id) when you POST to

/v1/chat/completions

. • The

llm_node

still uses

chat_ctx

for multi-turn textual history; audio is an additional payload tied to the latest user message. 6) Maintain chat_ctx alignment • When final transcript arrives, append the final STT text to

chat_ctx

as the user message (this keeps UI and history consistent). • When LLM returns assistant text, append assistant message to

chat_ctx

. • If you want the audio referenced in

chat_ctx

, add a metadata object or a short pointer in the user message (e.g.,

{ audio_ref: "<turn_id>" }

) so chat_ctx indicates which audio was used. 7) Concurrency & partial transcripts • Continue streaming interim transcripts to the UI as normal (the STT wrapper should forward interim events). • Only snapshot and send the buffered audio when the final-transcript/turn-complete event fires (avoids partial-audio mismatches). • If you need lower-latency partial audio sends, implement a policy in the STT wrapper to flush partial buffers with matching partial-transcript markers, but align finalization on the final transcript event. Why this pattern is clean and public-API friendly • Uses public STT node hook to capture raw audio — no manual room-track subscriptions or private internals. • Leaves STT provider unchanged for UI/transcript history. • Keeps

llm_node

simple: it consumes a clearly-signaled turn artifact rather than trying to intercept audio streams itself. • Keeps multi-turn

chat_ctx

semantics intact (STT user message appended first, assistant message appended after LLM reply). • Avoids requiring /v1/realtime or any private LiveKit APIs. Minimal code sketch (conceptual) • STT wrapper buffers frames, forwards frames to inner STT, on final event creates TurnArtifact {turn_id, audio_bytes, transcript} and stores it on the Agent (or passes to on_user_turn_completed hook). • llm_node checks Agent.pending_turn_artifact, converts audio_bytes if needed, POSTs to vLLM, streams text back, appends assistant message to chat_ctx, clears artifact. Final checklist to confirm • Do you want audio sent only after final transcript, or do you need partial/streaming audio to vLLM? • What audio format does vLLM expect (PCM, WAV, opus)? I can show the exact buffer-to-bytes conversion for that format. • Which STT provider are you using (so I can show the precise event names / helper functions)? If you want, I’ll provide a compact Python code example for the STT wrapper + turn artifact + llm_node wiring for your exact STT provider and vLLM audio format. Sources: Pipeline nodes and hooks | LiveKit Documentation | Capturing the First Few Seconds of a Call for Processing, with interactive elements

Can you now make this concrete for LiveKit Agents Python using exact public APIs? We want: • standard STT provider still handles transcripts/UI/history; • a wrapper around that STT path buffers the current user turn audio; • on final STT transcript, the wrapper creates a

TurnArtifact

containing: ◦

turn_id

◦ final transcript ◦ audio bytes or frames for that turn ◦ timestamps / sample rate if needed • custom

llm_node

reads the matching `TurnArtifact`; •

llm_node

calls

/v1/chat/completions

with: ◦ selected

chat_ctx

◦ current turn audio ◦ final STT transcript • assistant text streams to the existing TTS node. Please answer with exact Python SDK mechanisms: 1. What class/function should we wrap or override for STT? 2. Does the STT node/wrapper receive `rtc.AudioFrame`s directly? If yes, what is the method signature? 3. How do we forward the same frames to the underlying STT provider without breaking normal transcript behavior? 4. What exact event/object indicates interim transcript vs final transcript? 5. What exact public hook should create/store the

TurnArtifact

? 6. How can the custom

llm_node

access that

TurnArtifact

? 7. What is the exact Python signature for overriding

llm_node

? 8. In

llm_node

, how do we stream assistant text chunks back so LiveKit TTS consumes them? 9. Who commits the final user transcript into `chat_ctx`: the standard STT lifecycle or our wrapper? 10. Who commits the assistant final text into `chat_ctx`: LiveKit LLM/TTS lifecycle or our

llm_node

? 11. What public APIs should we use to avoid private methods like

_conversation_item_added

or direct

_chat_ctx

mutation? 12. Please provide a compact code skeleton showing: ◦ STT wrapper ◦

TurnArtifact

storage ◦ custom

llm_node

◦ AgentSession wiring Please mark any part that is approximate or version-dependent.

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!

Thanks — the architecture makes sense. Can you now make this concrete for LiveKit Agents Python using exact public APIs? We want: • standard STT provider still handles transcripts/UI/history; • a wrapper around that STT path buffers the current user turn audio; • on final STT transcript, the wrapper creates a

TurnArtifact

containing:

turn_id

final transcript audio bytes or frames for that turn timestamps / sample rate if needed • custom

llm_node

reads the matching `TurnArtifact`; •

llm_node

calls

/v1/chat/completions

with: selected

chat_ctx

current turn audio final STT transcript • assistant text streams to the existing TTS node. Please answer with exact Python SDK mechanisms: 1. What class/function should we wrap or override for STT? 2. Does the STT node/wrapper receive `rtc.AudioFrame`s directly? If yes, what is the method signature? 3. How do we forward the same frames to the underlying STT provider without breaking normal transcript behavior? 4. What exact event/object indicates interim transcript vs final transcript? 5. What exact public hook should create/store the

TurnArtifact

? 6. How can the custom

llm_node

access that

TurnArtifact

? 7. What is the exact Python signature for overriding

llm_node

? 8. In

llm_node

, how do we stream assistant text chunks back so LiveKit TTS consumes them? 9. Who commits the final user transcript into `chat_ctx`: the standard STT lifecycle or our wrapper? 10. Who commits the assistant final text into `chat_ctx`: LiveKit LLM/TTS lifecycle or our

llm_node

? 11. What public APIs should we use to avoid private methods like

_conversation_item_added

or direct

_chat_ctx

mutation? 12. Please provide a compact code skeleton showing: STT wrapper

TurnArtifact

storage custom

llm_node

AgentSession wiring Please mark any part that is approximate or version-dependent.