https://livekit.io logo
Join Slack
Powered by
I was testing out an agent locally, ocassionally, ...
# ask-ai
a
I was testing out an agent locally, ocassionally, the agent would start voicing the thinking tokens as well. I wonder what the issue could be? I am using gemini flash model and happenign with multiple voice providers
t
Thanks for reaching out, Deep. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — that sounds frustrating. You mentioned the agent sometimes speaks the model’s thinking tokens while using Gemini Flash and multiple voice providers. I can help narrow this down and give fixes that stop the thinking content from being sent to TTS. Why this happens (likely causes) • Gemini Flash supports a thinking mode and, by default, thinking content can be forwarded like other transcripts. If thinking output is enabled, those tokens (e.g., <think>…</think> or similar markers) can be included in the streamed content and then sent to your TTS provider. • If your code doesn’t filter or strip thinking tokens before passing text to your TTS pipeline, the TTS engine will vocalize them. • Intermittent behavior can come from model responses that sometimes include thoughts and sometimes don’t, or from thinking configuration not being consistently applied when you create/update the realtime model session. What you can do right now (practical fixes) 1. Disable thinking in the model session: ◦ Set the thinking configuration to exclude thoughts. For example:
Copy code
from google.genai import types

   session = AgentSession(
       llm=google.realtime.RealtimeModel(
           thinking_config=types.ThinkingConfig(
               include_thoughts=False,
           ),
       ),
   )
This prevents the model from forwarding thought content. 1. Filter thinking tokens before TTS (defensive approach): ◦ Intercept and clean the LLM stream output and remove tags like 
<think>
and 
</think>
(or any other thought markers) before sending text to the TTS provider. ◦ Example approach (stream processing):
Copy code
async for chunk in stream:
       content = getattr(chunk.delta, 'content', None) if hasattr(chunk, 'delta') else str(chunk)
       if content:
           content = content.replace("<think>", "").replace("</think>", "")
           # send cleaned content to TTS
1. Create a custom Agent 
llm_node
to modify output: ◦ Override the LLM output node so all streamed chunks are cleaned of thinking tokens and any internal markers before they reach the voice pipeline. Questions to pinpoint the root cause • Which exact Gemini model are you using (for example 
gemini-2.5-flash
or 
gemini-2.5-flash-native-audio-preview-12-2025
)? • Are you using the realtime Gemini Live API (realtime model) or a standard LLM endpoint? • Do you currently set any 
thinking_config
options? If so, what are they? • Can you confirm the exact text markers you’re seeing in the spoken output (e.g., literal 
<think>
tags, “(thinking)”, or free-form internal notes)? • Is the behavior truly intermittent for the same prompt, or does it correlate with certain prompts or tools? If you share the model name, a snippet showing how you create the model/session, and an example of the spoken output (the raw text before TTS if possible), I’ll suggest the exact code changes to stop thinking tokens from being vocalized. Sources: Gemini Live API plugin | LiveKit Documentation | Modifying LLM output before TTS | LiveKit Documentation
a
Copy code
async def entrypoint(ctx: JobContext):
    await ctx.connect()

    agent = Agent(
        instructions=INSTRUCTIONS,
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-3", language="multi"),
        llm=google.LLM(model="gemini-2.5-flash",
          vertexai=True
        ),
        tts=sarvam.TTS(
            target_language_code="hi-IN",
            speaker="shubh",
            model="bulbul:v3",
            pace=1.1,
        ),
    )

    session = AgentSession()
    await session.start(agent=agent, room=ctx.room)
    await session.say("Hello?", allow_interruptions=True)
HEre is my code, i dont want a real time model, and idont think livekit requires me to handle thinking token it should handle thinking token handling out of box in the gemini plugin?
t
Thanks — this is very helpful. I looked at your code and your question about whether the Gemini plugin handles thinking tokens automatically for non-realtime models. I’ll answer directly and give the minimal changes you can make to stop thinking tokens from being spoken. Short answer • Gemini Flash can produce thinking output. In your current setup (you’re using 
google.LLM(model="gemini-2.5-flash", vertexai=True)
— not a realtime model), thinking output is not automatically filtered out for you. That means if the model emits thought markers (for example 
<think>...</think>
or similar internal text), those tokens can reach your TTS unless you explicitly prevent or remove them. What you can do right now (2 recommended options) 1) Disable thinking in the model session (preferred if available) • If your LLM wrapper supports a 
thinking_config
or similar option, set it to not include thoughts. This prevents the model from forwarding thought content into responses. 2) Filter model output before sending to TTS (defensive and reliable) • Intercept the text returned by the LLM and remove thought markers before passing text to 
tts
. • Example approach (pseudo/async pattern matching your code style):
Copy code
# after getting model text (sync or streamed), remove markers
clean_text = raw_text.replace("<think>", "").replace("</think>", "")
# send clean_text to TTS
• If your LLM returns streaming chunks, perform this cleaning per chunk before forwarding to the TTS pipeline. Why the behavior is intermittent • The model only produces thinking content for some responses, so you’ll see the TTS speak thoughts only when the model emits them. • Not using a realtime model does not automatically guarantee that thinking tokens won’t be present; they must be suppressed or removed. Concrete next steps I recommend 1. Confirm whether your 
google.LLM
wrapper supports a 
thinking_config
(or equivalent). If it does, set 
include_thoughts=False
when constructing the LLM/session. 2. Add a simple post-processing step between LLM output and your 
tts
call to strip out any 
<think>
or other internal markers. 3. If you want, I can give a small, concrete code snippet that plugs into your current 
entrypoint
flow showing where to filter the LLM output (non-realtime case). Do you receive the LLM output as a single string, or do you handle streaming chunks in your code? If you share whether your 
google.LLM
object has any 
thinking_config
option or how your code receives the LLM response (single text vs stream), I’ll provide the exact code you can paste into your project.
a
google.LLM is a livekit pluging the only code i have written is sent above, so you tell me what i need ot change to avoid this.
t
Thanks — this is clear and helpful. I’ll address what you need to change in your entrypoint so Gemini Flash stops sending thinking tokens to TTS. Short answer • LiveKit’s 
google.LLM
plugin (non‑realtime) does not automatically strip thinking content for you. You need to prevent thinking tokens either by configuring the model to exclude thoughts (if the plugin exposes that option) or by filtering the LLM output before calling TTS. Since you only control the code you shared, the reliable fix is to filter the model output before passing it to 
tts
. Two practical options (pick one) 1) Preferred — disable thinking in the model if the plugin exposes a thinking config • If the LiveKit 
google.LLM
plugin supports a 
thinking_config
or similar parameter, construct the LLM with thoughts disabled:
Copy code
llm = google.LLM(
    model="gemini-2.5-flash",
    vertexai=True,
    thinking_config={"include_thoughts": False}  # if supported by the plugin wrapper
)
agent = Agent(
    instructions=INSTRUCTIONS,
    vad=silero.VAD.load(),
    stt=deepgram.STT(model="nova-3", language="multi"),
    llm=llm,
    tts=sarvam.TTS(...),
)
• If that option exists in your LiveKit wrapper, this prevents the model from emitting thought content at the source. 2) Reliable — strip thinking markers right before you call TTS (works regardless of wrapper support) • Post‑process the text returned by the LLM and remove common thinking tokens (for example 
<think>...</think>
) before sending to 
session.say()
or your TTS pipeline. • Minimal change example that fits your entrypoint pattern. Insert a small helper and use it when you want the agent to speak:
Copy code
def strip_thoughts(text: str) -> str:
    if not text:
        return text
    # remove common thought markers; extend if you see different markers
    return text.replace("<think>", "").replace("</think>", "").replace("(thinking)", "")

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    agent = Agent(
        instructions=INSTRUCTIONS,
        vad=silero.VAD.load(),
        stt=deepgram.STT(model="nova-3", language="multi"),
        llm=google.LLM(model="gemini-2.5-flash", vertexai=True),
        tts=sarvam.TTS(
            target_language_code="hi-IN",
            speaker="shubh",
            model="bulbul:v3",
            pace=1.1,
        ),
    )

    session = AgentSession()
    await session.start(agent=agent, room=ctx.room)

    # Example: clean text before sending to be spoken
    raw = "Hello?"  # if you get raw from LLM, apply strip_thoughts(raw)
    cleaned = strip_thoughts(raw)
    await session.say(cleaned, allow_interruptions=True)
If your speech flow uses streamed chunks (the agent streaming model output into TTS), apply 
strip_thoughts()
to each chunk before forwarding to the TTS call. What I need from you to give an exact patch • Does the LiveKit 
google.LLM
constructor accept a 
thinking_config
or 
thinking_config={"include_thoughts": False}
? If you’re not sure, tell me the exact LiveKit/Google plugin version or paste the attributes available on 
google.LLM
in your environment and I’ll confirm the right parameter name. • Do you receive the LLM output as a single string (synchronous) or as streaming chunks in your current agent flow? If streaming, paste the small snippet where you receive/send those chunks and I’ll show precisely where to filter. Next steps I can help with • Provide a one-line patch for your current integration if you confirm whether 
thinking_config
exists. • Give a streaming-safe filter snippet if your flow uses chunked output. If you’d like, paste the exact spoken output you see (the raw text before TTS) so I can add any additional markers to strip.
2 Views
Open in Slack
PreviousNext
Powered by