https://livekit.io logo
Join Slack
Powered by
how does the flow in livekit work. How does stt mo...
# ask-ai
c
how does the flow in livekit work. How does stt model pass the transcribed text to llm. Explain the flow in detail with code reference
✅ 1
t
Thanks for reaching out, Kushagra. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’ll explain the LiveKit voice-agent flow and exactly how STT output is delivered to the LLM, then show the key code references and minimal snippets so you can follow the plumbing. Acknowledging your request: you asked how the flow works end-to-end and how the transcribed text from STT is passed to the LLM, with code references. High-level flow (streaming, low-latency) • Audio from a LiveKit room → Voice Activity Detection (VAD) segments speech. • VAD-segmented audio frames go to the STT node (streaming transcription). • STT emits partial and final transcripts as transcript events. • Transcript events are wrapped into the agent’s conversation context (ChatContext / SpeechEvent). • The AgentSession routes those transcript events into the LLM node as input messages. • LLM streams generated tokens back; those tokens can be sent to TTS for audio or returned as text. Where this is implemented • The pipeline is managed by an AgentSession which declares model nodes (STT, LLM, TTS, VAD). See the Voice AI quickstart for example usage. • Pipeline nodes and hooks handle the event flow: the STT node emits SpeechEvents and the LLM node consumes them. See Pipeline nodes and hooks for node behavior and hook points. • The sequential/streaming architecture is described in the Sequential Pipeline Architecture and Voice Agent Architecture posts. Key code references and minimal snippets 1) Declare session with STT and LLM (example pattern)
Copy code
session = AgentSession(
    stt=inference.STT(model="deepgram/nova-3", language="multi"),
    llm=inference.LLM(model="openai/gpt-5.2-chat-latest"),
    tts=inference.TTS(model="cartesia/sonic-3", voice="..."),
    vad=silero.VAD.load(),
)
• This declarative config wires the STT and LLM into the same session pipeline so STT output flows into the LLM. 2) Start the session and attach to a room
Copy code
await session.start(
    agent=Assistant(),
    room=ctx.room,
    room_options=room_io.RoomOptions(...)
)
• Once started, the session listens to the room audio track, VAD segments it, and sends audio to the STT node. 3) STT node → emits transcript events (conceptual) • STT produces streaming partial results and final transcripts as SpeechEvent/Transcript objects. • Those events are propagated into the session’s ChatContext. The LLM node subscribes to those events. 4) LLM consumption (conceptual) • The LLM node receives the transcript event (current turn + chat history) and produces streaming tokens. • Tokens are available to downstream nodes (e.g., TTS) as they are produced. Helpful implementation notes • STT streaming reduces latency by sending partial transcripts to the LLM as they arrive. • The ChatContext object keeps conversation state so the LLM has history with each transcript. • Pipeline nodes and hooks let you intercept transcripts if you need custom preprocessing, moderation, or enrichment before sending to the LLM. Links for code and deeper details • Voice AI quickstart: https://docs.livekit.io/agents/start/voice-ai/ • Pipeline nodes and hooks: https://docs.livekit.io/agents/logic/nodes/ • Sequential pipeline architecture (design + streaming behavior): https://livekit.com/blog/sequential-pipeline-architecture-voice-agents • Voice agent architecture (STT→LLM→TTS explained): https://livekit.com/blog/voice-agent-architecture-stt-llm-tts-pipelines-explained Next step for me to help further • Do you want a concrete, runnable example (full Python file) wired to a specific STT provider and an LLM provider? If so, tell me which STT and LLM you plan to use and whether you want streaming partials handled or only final transcripts. Sources: Sequential Pipeline Architecture for Voice Agents | LiveKit | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Voice AI quickstart | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit, with interactive elements
c
what if i want to alter the transcribed text before passing it to llm
t
Thanks — I’m responding to your question about altering STT transcripts before they reach the LLM. Short answer • Override the STT pipeline node in your Agent (the 
stt_node
/ 
sttNode
hook), modify the 
SpeechEvent
(or its 
alternatives[0].text
) and yield the changed events; the session will pass those events on to the LLM node. Why this works • The pipeline uses nodes/hooks: the STT node emits SpeechEvent objects, and the LLM node consumes them. Intercepting the STT node lets you run text normalization, filtering, enrichment, or moderation before the LLM sees it. Minimal examples Python (async generator)
Copy code
async def stt_node(self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings) -> Optional[AsyncIterable[stt.SpeechEvent]]:
    # optionally preprocess audio here...
    events = Agent.default.stt_node(self, audio, model_settings)

    async for event in events:
        if event.type == stt.SpeechEventType.FINAL_TRANSCRIPT:
            # modify the top alternative text
            event.alternatives[0].text = event.alternatives[0].text.replace("um", "")
            # or apply normalization, profanity filter, enrichment, etc.
        yield event
TypeScript / JS (ReadableStream)
Copy code
async sttNode(audio, modelSettings) {
  const events = await voice.Agent.default.sttNode(this, audio, modelSettings);
  if (!events) return null;

  return new ReadableStream({
    start(controller) {
      const reader = events.getReader();
      const pump = async () => {
        const { done, value } = await reader.read();
        if (done) { controller.close(); return; }
        if (value.type === 'FinalTranscript') {
          value.alternatives[0].text = value.alternatives[0].text.toLowerCase();
        }
        controller.enqueue(value);
        pump();
      };
      pump();
    }
  });
}
Practical patterns to consider • Drop filler words / normalize casing / expand contractions • Apply profanity or PII redaction • Run entity recognition and tag the transcript (add metadata fields) • Detect intents or keywords and add system messages to ChatContext • Keep partial transcripts intact if you want low-latency streaming to LLM; only modify final transcripts if you need more reliable text Next step • Do you want a full runnable example for Python or TypeScript tied to a specific STT/LLM provider, or a sample that shows inserting moderation or enrichment logic? Sources: Pipeline nodes and hooks | LiveKit Documentation, with interactive elements
c
can i access the userdata here
t
Yes — you can access session 
userdata
while modifying STT transcripts before they go to the LLM. Acknowledging your question: you want to read/use user-specific data during the STT → LLM preprocessing step. How to access userdata • In your custom Agent, 
self.session.userdata
(Python) or 
this.session.userData
(TypeScript) is available inside 
stt_node
/ 
sttNode
. • Use that object to drive normalization, filtering, personalization, routing, or logging before yielding the modified 
SpeechEvent
. Minimal examples Python
Copy code
class MyAgent(Agent):
    async def stt_node(self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings):
        userdata = self.session.userdata  # access userdata
        events = Agent.default.stt_node(self, audio, model_settings)
        async for event in events:
            if event.type == stt.SpeechEventType.FINAL_TRANSCRIPT and event.alternatives:
                text = event.alternatives[0].text
                # example: personalize based on userdata
                if getattr(userdata, "preferred_language", None) == "es":
                    text = translate_to_spanish(text)  # your function
                event.alternatives[0].text = normalize(text)
            yield event
TypeScript / Node
Copy code
class MyAgent extends voice.Agent<MyUserData> {
  async sttNode(audio, modelSettings) {
    const userData = this.session.userData; // access userdata
    const events = await voice.Agent.default.sttNode(this, audio, modelSettings);
    if (!events) return null;

    return new ReadableStream({
      start(controller) {
        const reader = events.getReader();
        const pump = async () => {
          const { done, value } = await reader.read();
          if (done) { controller.close(); return; }
          if (value.type === 'FinalTranscript' && value.alternatives) {
            let text = value.alternatives[0].text;
            if (userData.preferredLanguage === 'es') {
              text = translateToSpanish(text); // your function
            }
            value.alternatives[0].text = normalize(text);
          }
          controller.enqueue(value);
          pump();
        };
        pump();
      }
    });
  }
}
How to ensure userdata is present • Provide userdata when creating the AgentSession: ◦ Python: 
AgentSession(userdata=MySessionInfo(user_id="u1", preferred_language="es"), ...)
◦ TypeScript: 
new voice.AgentSession({ userData: { userId: 'u1', preferredLanguage: 'es' }, ... })
Practical tips • Keep userdata reads fast (avoid blocking I/O) to preserve streaming latency. • Use userdata for personalization (language, tone), routing, moderation flags, or adding system messages to ChatContext. • If userdata can change at runtime, read it per event (as shown) rather than caching once. Would you like a runnable example wired to a specific STT/LLM provider and a sample userdata shape? Sources: Pipeline nodes and hooks | LiveKit Documentation | Agents and handoffs | LiveKit Documentation, with interactive elements
2 Views
Open in Slack
PreviousNext
Powered by