https://livekit.io logo
Join Slack
Powered by
how realtime llm model updates chat_ctx?
# ask-ai
w
how realtime llm model updates chat_ctx?
t
Thanks for reaching out, Test. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about how a real-time LLM updates 
chat_ctx
. Short answer • A real-time LLM continuously updates 
chat_ctx
as it processes streaming audio and generates streaming responses. It incrementally adds user and assistant messages to the conversation state rather than waiting for discrete turns. How that works in practice • Streaming audio in: The model accepts raw audio frames (not only finalized full utterances). As audio arrives, the model increments its internal representation of the user’s input. • Incremental context updates: The model appends partial and then finalized user content to 
chat_ctx
as speech is recognized and interpreted. This keeps the conversation state up to date even with overlaps or interruptions. • Streaming output: The model can emit streamed text/audio responses while the context keeps evolving, so replies can be generated with awareness of immediately preceding audio. • Lifecycle hooks and turn events: When the model detects a turn boundary (for example, transitions like “listening” → “speaking” or a completed user turn), pipeline hooks such as 
on_user_turn_completed
can run custom logic to update 
chat_ctx
further, add summaries, or inject system messages. • Programmatic modifications: You can also modify 
chat_ctx
programmatically (for example with an 
update_chat_ctx()
method or by injecting system/context messages) if you need to enforce policy, add metadata, or replace portions of the generated output before TTS. When you might choose programmatic updates • You want to insert a short contextual summary to keep very long conversations efficient. • You need to enforce or sanitize content before it becomes part of the visible assistant messages. • You want to override or replace LLM output before sending to TTS. If you want next steps • Do you want an example of how to call 
update_chat_ctx()
or where to add 
on_user_turn_completed
logic in a LiveKit pipeline? • Or tell me whether you’re using a boxed real-time model (e.g., OpenAI Realtime / Gemini Live) or a custom pipeline so I can tailor the example. Sources: Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Modifying LLM output before TTS | LiveKit Documentation | LLM Output Replacement | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation
w
yes how to call 
update_chat_ctx with realtime model
t
Thanks — I’m responding to your request for a concrete example of how to call 
update_chat_ctx()
when using a realtime LLM model. Below is a concise example and notes showing where to call 
update_chat_ctx()
from inside an agent that uses a realtime model (the method works the same as with non-realtime models). The example assumes an async agent lifecycle (typical in LiveKit agent code). Example (Python, inside an Agent class)
Copy code
from livekit.agents import ChatContext, ChatMessage

class Assistant(Agent):
    async def on_user_turn_completed(self, ctx):
        # Build or modify the chat context you want the model to see
        new_ctx = ChatContext()
        new_ctx.add_message(role="user", content="Injected context: user prefers brief answers.")
        new_ctx.add_message(role="system", content="Please prioritize concise responses.")

        # Update the active chat context for the realtime model
        await self.update_chat_ctx(new_ctx)

        # Optionally continue with other logic (e.g., trigger a response)
        # await self.generate_response()   # pseudocode
Key points to keep in mind • The method is invoked on the agent instance (e.g., 
await self.update_chat_ctx(...)
) and accepts a 
ChatContext
containing text messages. • Realtime models accept audio input directly, but 
chat_ctx
itself is text-only — injected context must be text. • Typical places to call `update_chat_ctx()`: ◦ After 
on_user_turn_completed
or similar lifecycle hooks when a logical turn finishes. ◦ When you want to inject summaries, policy/system messages, or corrected/sanitized text before the model replies. • You can both add messages (as shown) or replace the entire context depending on your needs. If you’d like, I can: • Provide a full runnable snippet including realtime model session setup (Gemini Live / OpenAI Realtime) and where to place this agent, or • Show an example that updates context conditionally (e.g., only when conversation length exceeds N messages) — which one would help you most? Sources: Vision | LiveKit Documentation | Realtime models overview | LiveKit Documentation
Open in Slack
PreviousNext
Powered by