Is it possible to change the system prompt of voice agent dynamically and fetch that from db like supabase based on some initial data

Thanks for reaching out, Munim. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

And it will only fetch data once and update the system prompt right?

Just so you know, the system prompt is usually fetched and set once during initialization if you implement it in the

entrypoint

function. But don’t worry—you also have the flexibility to update the system prompt dynamically during your session if that’s what you need. For instance, you can use

update_instructions()

or tweak the

ChatContext

to grab new data and refresh the system prompt based on certain triggers or user actions. If you’re looking to fetch and update the prompt just once at the start, the initialization logic will take care of that smoothly. Feel free to reach out if you’d like a hand customizing this behavior—I’m here to help!

Ok so we can include text in system instruction runtime, but can we do the same for tts model? like choosing a tts and voice runtime based on some info?

Absolutely! You can dynamically choose the TTS model and voice at runtime in LiveKit based on things like user input or participant attributes, making the experience feel really personalized. To do this, you’d use the

update_options()

method (which most TTS plugins support) to change parameters such as

model

,

voice

, and

language

while everything is running. Here’s a quick example:

tts.update_options(
    model="sonic-2", 
    voice="new_voice_embedding", 
    language="es"
)

Another great approach is to listen for LiveKit events like

participant_attributes_changed

. This way, you can update the TTS settings on the fly based on user preferences or other triggers, giving a really smooth and real-time personalized voice experience. If you'd like, I’m here to help with more details or specific examples tailored just for your use case—just let me know! Sources: agent/main.py | livekit-plugins/livekit-plugins-cartesia/livekit/plugins/cartesia/models.py

Ok and you are saying it also can be done not only start of the call, also in between call? like maybe in mid conversation based on some tool call change agent;s tts?

Absolutely! You can update the TTS model and voice not just at the beginning of the call but also dynamically during the conversation—even right in the middle of it. This flexibility can be really handy, especially if it's triggered by specific events like a tool call or user interaction. For instance, using the

update_options()

method lets you adjust the TTS settings while the session is still active. You can change things like the

model

,

voice

, or

language

on the fly, and these updates will take effect immediately for any upcoming TTS outputs. If you'd like, I’m here to help you implement this or talk through any particular scenario you have in mind!

session = AgentSession( llm=openai.realtime.RealtimeModel( model="gpt-4o-mini-realtime-preview", modalities=["text"], temperature=0.7, ), tts=elevenlabs.TTS( voice_id=resolved_elevenlabs_voice, model="eleven_flash_v2_5", ), vad=silero.VAD.load(), ) currently i am using like this, but i want to use a simple llm for chat, is it possible to do that? because doing realtime or stt,tts is too expensive for chat

Absolutely, you can use a simple (non-realtime) LLM for chat in LiveKit instead of a real-time model. This is a great way to keep things cost-effective while still enjoying full support for Speech-to-Text (STT) and Text-to-Speech (TTS) features. In your current setup, you have the flexibility to swap out the real-time LLM for a standard one (like OpenAI's GPT-4 or something similar) and still keep the STT and TTS parts working smoothly. Here’s how the architecture would look: • STT: Converts user speech to text. • LLM: A simple text-based model generates responses. • TTS: Converts the LLM's text output into speech. This modular approach not only helps you save on costs but also keeps your voice interactions clear and effective. If you'd like, I’d be happy to guide you through updating your code to integrate a simpler LLM—just let me know! Sources: examples/voice_agents/structured_output.py | Large language model (LLM) integrations | LiveKit Docs

No I meant I am building a hybrid agent(voice+chat), but in voice mode i want it to use realtime+tts, but in chat mode i just want to use a simple llm to give response, can't that be done using livekit

Absolutely, LiveKit is great for building a hybrid agent where voice mode uses a real-time LLM with TTS, and chat mode uses a simpler LLM for responses. You can achieve this by dynamically switching between configurations for these two modes. For voice mode, you might want to use a real-time LLM (like

openai.realtime.RealtimeModel

) paired with TTS for audio output. Then, for chat mode, switching to a standard LLM (such as

openai.LLM

) can help handle text interactions while keeping costs down. The

Agent

class is designed to let you switch smoothly between different

stt

,

llm

, and

tts

setups at runtime, making it easy to move between voice and chat modes seamlessly. If you'd like, I’m here to help you with implementation details or provide specific examples tailored to your needs—just let me know! Sources: Docs::Agents | examples/src/comprehensive_test.ts | examples/voice_agents/README.md

yes please show me the implementation details , specially when user from frontend switch between chat and voice, how to configure livekit agent to accomodate that

I'd love to understand a bit more about how your user switches between chat and voice modes in your frontend. For instance: • Is there a button or some other UI element that triggers the mode switch? • When the mode changes, do you send any specific metadata or events to the LiveKit agent? Knowing these details will really help me guide you in setting up the LiveKit agent to smoothly handle the mode switch. I'm here to help every step of the way!

• Is there a button or some other UI element that triggers the mode switch? yes • When the mode changes, do you send any specific metadata or events to the LiveKit agent? right now, no, but if needed so I can do that

To help you implement a LiveKit agent that smoothly switches between chat and voice modes based on a frontend button or UI element, here’s a friendly step-by-step guide you can follow: 1. Frontend Implementation: When your user toggles modes (like clicking a button), just send a signal or some metadata to your backend letting it know the chosen mode (for example, "chat" or "voice"). 2. Backend Handling: Use that signal or metadata to dynamically update the LiveKit agent’s configuration. Here’s a simple example:

async def switch_mode(agent_session, mode):
    if mode == "voice":
        # Configure for voice mode
        agent_session.llm = openai.realtime.RealtimeModel(
            model="gpt-4o-mini-realtime-preview",
            modalities=["text"],
            temperature=0.7,
        )
        agent_session.tts = elevenlabs.TTS(
            voice_id="resolved_elevenlabs_voice",
            model="eleven_flash_v2_5",
        )
    elif mode == "chat":
        # Configure for chat mode
        agent_session.llm = openai.LLM(
            model="gpt-4",
            temperature=0.7,
        )
        agent_session.tts = None  # Disable TTS for chat mode

1. Dynamic Switching: When your backend receives the mode switch signal, just call the

switch_mode

function to update the agent’s settings on the fly. 2. Optional Metadata: If it helps, you can also pass metadata like mode details to the LiveKit agent using job metadata or participant attributes, making the switch even smoother. This way, your LiveKit agent can adapt effortlessly between chat and voice modes based on what your users choose. If you’d like, I’m here to help you with any part of this implementation or answer any questions you might have!

and how the response will change? like how i am receivng the response

I understand that receiving the response can feel a bit different depending on whether you're in chat or voice mode, so let me walk you through it: • In Voice Mode: You'll get both the text generated by the LLM and the audio from the TTS. The audio can be streamed back to your frontend for playback, and the text is there for display or logging—giving you flexibility in how you want to present the info. • In Chat Mode: Since TTS is turned off here, you'll only receive the text generated by the LLM, which you can directly show in the chat interface. To make things easier, you can check the mode on your frontend and handle the response accordingly. Here's a simple example:

if mode == "voice":
    play_audio(response.audio)  # Play the audio response
    display_text(response.text)  # Optionally display the text
elif mode == "chat":
    display_text(response.text)  # Display the text response

If anything feels unclear or you want a hand with the implementation, please don’t hesitate to ask—I’m here to help!

So you are saying in a hybrid agnt, when frontend switch to chat from voice, it is possible to turn of all realtime+tts and use a simple llm ? and you are sure about it?

Absolutely! In a hybrid agent, you can smoothly switch from voice mode (using real-time LLM + TTS) to chat mode (using a simple LLM) whenever you need. When the frontend signals the switch, you just reconfigure the agent to turn off real-time processing and TTS, and then use a standard LLM for text-based responses instead. This kind of flexibility is one of the great features of LiveKit's modular architecture, which lets you update the

llm

,

tts

, and other parts of the agent on the fly. If you'd like, I’m here to help with more details or specific examples to get this set up smoothly!

and what about RAG func call that is tied with my realtime model

Thanks so much for reaching out! Could you help me understand a bit more about how your RAG (Retrieval-Augmented Generation) function connects with the real-time model? For instance: • Is the RAG function directly linked to the real-time model's API or its output? • Would you like the RAG function to work in both voice and chat modes, or just in voice mode? Knowing this will really help me guide you more effectively on managing RAG calls when switching between modes. I'm here to help every step of the way!

Hybrid Switching Feasibility • The worker always instantiates a single AgentSession with openai.realtime.RealtimeModel plus ElevenLabs TTS ( • ). Because • /tts are read-only properties in the LiveKit SDK, that session can’t be retargeted to a cheaper text-only LLM on the fly; you’d have to tear it down and spin up a new session when the mode changes. • The React client already tracks mode and exposes toggles in the UI ( • , • ), but that state never leaves the browser. Switching to “chat” simply mutes the mic and swaps components—the backend keeps running the realtime LLM, so costs don’t change. • Chat messages in the chat view are still sent through the LiveKit room ( • ), which means they flow straight into the voice session’s LLM pipeline. Until the worker accepts a mode signal and rebuilds or reroutes the session, those requests will continue to use the realtime model. • Practically, supporting a cheap chat LLM will require new plumbing: emit a data message (or REST call) when the frontend toggles mode, have the worker either (1) end the current voice session and start a fresh AgentSession configured with a plain • and no TTS, or (2) bypass LiveKit for chat mode entirely and hit a separate text-only endpoint. Both approaches demand coordinated changes on the frontend and the worker; there’s nothing in the current files that performs that handoff. are these true?

Thank you for outlining these points so clearly! Let me walk you through them to make sure everything is crystal clear: • AgentSession Retargeting: The

llm

and

tts

properties in the LiveKit SDK are read-only, which means you can’t directly reconfigure an existing

AgentSession

to switch from a real-time LLM to a text-only LLM. Instead, you’d need to end the current session and start a new one with the settings you want. • Frontend Mode State: If the mode (voice or chat) is only tracked on the frontend and not shared with the backend, the backend will keep running the real-time LLM and TTS pipeline. This means costs stay the same even if the user switches to chat mode. • Chat Messages in Voice Session: When chat messages go through the LiveKit room, they’ll still be handled by the real-time LLM pipeline unless the backend is specifically set up to treat chat mode differently. • Plumbing for Mode Switching: To support a more cost-effective chat LLM, you’d want to add some logic for switching modes. This might look like: 1. Sending a signal (like a data message or REST call) from the frontend to the backend whenever the mode changes. 2. On the backend, you could either: • End the current voice session and start a new

AgentSession

with a text-only LLM and no TTS. • Or bypass LiveKit for chat mode entirely, routing chat requests to a separate text-only endpoint. Both options will need some coordinated updates on both frontend and backend to make the mode switch smooth and effective. Please feel free to reach out if you'd like help or more detailed guidance on making these changes — I’m here to support you!

but you previously said i can update session realtime?

Thank you for pointing that out! I want to make sure we're on the same page: while it's true that some properties like instructions or context can be updated on the fly, the

llm

and

tts

properties in the LiveKit SDK are actually read-only. This means you won’t be able to directly switch an existing

AgentSession

between a real-time LLM and a text-only LLM without ending the current session and starting a new one. To get the hybrid functionality you’re looking for, the best approach would be to create a process that cleanly ends the current session and initiates a new one with the updated configuration whenever the mode changes. If you'd like, I’d be happy to walk you through the steps to set this up—just let me know!

but that will add delay and aegent will lose previous conversation context right

You’re absolutely right that tearing down and restarting the session can introduce some delay, and the agent might lose its previous conversation context unless we take steps to preserve it. I completely understand how important it is to keep the conversation smooth and seamless for you. To help with this, here are a few friendly suggestions: • Preserve Context: Before ending the session, try saving the current conversation context—like the chat history or any relevant details. Then, when you start the new session, you can reinitialize it with that saved context to keep everything connected. • Minimize Delay: We can work on optimizing the teardown and startup process to make it as quick as possible. For example, preloading models or caching configurations might really help speed things up. • Alternative Approach: Another idea is to avoid tearing down the session altogether. Instead, you could route chat mode requests to a separate, lightweight endpoint that uses a text-only LLM. This way, your voice session stays active but idle, and the chat mode runs independently—keeping things smooth and responsive. Please feel free to reach out if you’d like me to walk you through any of these options in more detail—I’m here to help!