https://livekit.io logo
Join Slack
Powered by
I recently updated to Gemini 3 flash for my llm in...
# ask-ai
f
I recently updated to Gemini 3 flash for my llm in my stt-llm-tts stack with thinking config set to low, and I'm for some reason seeing the thinking Gemini is doing fed into the output read out to the user. This never happened previously with Livekit 1.38 Are there any params to ensure additional safeguards against the llm thinking being fed into User facing content?
t
Thanks for reaching out, Akash. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I hear you: after upgrading to Gemini 3 Flash your STT→LLM→TTS pipeline is passing the model’s internal “thinking” (intermediate tags/tokens) into the user-facing audio, and you want parameters or safeguards to stop that. Short answer: there isn’t a global parameter that disables thinking for Gemini 3 Flash in a standard inference-based STT‑LLM‑TTS pipeline. The `thinking_config`/`include_thoughts=False` option only applies to the Gemini Live (realtime) API, not to standard inference calls. To prevent internal reasoning from reaching TTS, the recommended approach is to filter or sanitize the LLM output before feeding it to the TTS node. What you can do now (recommended, minimal changes) 1. Override your agent’s 
llm_node
(or insert an LLM output processing step) and strip/replace internal tags or tokens before passing text to TTS. 2. Specifically remove or replace tags like 
<tool_call>
, the word 
found
, or other chain-of-thought markers your model emits. 3. Test using short prompts and log the raw LLM stream to see exactly which markers to filter. Example pattern (conceptual):
Copy code
async def llm_node(self, chat_ctx, tools, model_settings=None):
    async with self.llm.chat(chat_ctx=chat_ctx, tools=tools) as stream:
        async for chunk in stream:
            content = getattr(chunk.delta, "content", None) if hasattr(chunk, "delta") else str(chunk)
            if content:
                content = content.replace("<tool_call>", "").replace("found", "")
                if hasattr(chunk, "delta") and hasattr(chunk.delta, "content"):
                    chunk.delta.content = content
            yield chunk
Why this changed after upgrading • Gemini 3 Flash (or its backend) can emit more verbose internal reasoning/tags. • Newer LiveKit versions may stream intermediate tokens differently, exposing them unless you sanitize the stream. Extra safeguards you can add • Centralized content filter node that enforces patterns (regex) or runs a brief safety LLM to rephrase/remove thinking traces before TTS. • Whitelist acceptable output formats and drop anything that looks like internal metadata. If you’d like, tell me: • an example raw model output you’re seeing (paste a short snippet), and • whether your pipeline uses streaming chunks (yes/no), and I’ll provide a specific filter snippet you can drop into your 
llm_node
to remove those exact tokens. Sources: Gemini Live API plugin | LiveKit docs | Google Gemini LLM | LiveKit docs | LLM Output Replacement | LiveKit docs | Modifying LLM output before TTS | LiveKit docs | LLM-Powered Content Filter | LiveKit docs
f
Outside of when it’s calling tools, how can I distinguish thinking vs output tokens appearing during the response?
t
Thanks — you’re asking how to tell thinking tokens apart from real output when the model isn’t calling tools. Short answer: it depends on which model/streaming API you’re using. • If you’re using the Gemini Live (realtime) API: thinking vs output are emitted as distinct event types/fields (e.g., 
thoughts
vs `delta`/`text`) and you can check that metadata in your 
llm_node
to drop thinking before TTS. • If you’re using standard inference with Gemini 3 Flash (non‑Live): there isn’t a guaranteed metadata flag that marks “thinking” tokens. Distinction relies on patterns in the text stream (delimiters/tags, token strings, or token sequencing) and on filtering logic you add in 
llm_node
. Practical, actionable options you can implement now 1. Log raw streamed chunks to see exact markers the model emits (examples like 
<tool_call>
, 
found
, or explicit delimiters). 2. In 
llm_node
, detect and drop thinking chunks by: ◦ Checking chunk fields for 
thoughts
/ 
part_type
/ 
role
when using Gemini Live; OR ◦ Applying text-based rules for inference streams: regex to remove known tags, drop lines inside delimiters, or drop segments matching chain-of-thought patterns. 1. Add a short classifier/whitelist before TTS: reject or rephrase anything not matching your allowed output format (e.g., must start with a response sentence, or must not contain 
[...]
or 
<tool_*>
tags). 2. If you get intermediate token streaming, buffer until you see a finalizing marker (e.g., model sends an end-of-response event) and only forward buffered text to TTS. If you share a short raw snippet of what you’re seeing in the stream (a few tokens/chunks that include the thinking text and the intended output), I’ll write a precise 
llm_node
filter or regex you can drop in to remove those thinking tokens. Sources: Gemini Live API plugin | LiveKit docs
f
Can you give me a high level of how thinking works in 3 flash non live especially dealing with outputs to tts
t
Thanks — I’ll give a concise, high-level explanation of how “thinking” typically behaves for non‑live Gemini-style models (like Gemini 3 Flash used via standard inference) and what that means for TTS output. High-level behavior • Non‑live inference models do not expose a built‑in, structured “thinking” stream the way Gemini Live does. There’s no guaranteed metadata flag that marks internal reasoning vs final text. • Internally the model may generate chain‑of‑thought tokens or intermediate scaffolding as part of producing the final response. With standard inference those may be concatenated into the text stream the model returns unless the model or prompt explicitly suppresses them. • Because the inference response is what you hand to TTS, any internal tokens/tags that appear in that returned text will be spoken unless you clean them first. How this shows up in a STT→LLM→TTS pipeline • Single-shot inference: the model returns a final string. If that string contains internal markers (e.g., special tags, bracketed reasoning, or explicit chain‑of‑thought text), TTS will synthesize it verbatim. • Streaming/incremental inference: partial tokens may arrive over time. Without filtering you can accidentally forward intermediate fragments to TTS, producing “thinking” audio. Practical implications / recommended safeguards • Treat the LLM output as untrusted for user‑facing speech — sanitize it before TTS. • Buffer vs stream: prefer buffering until a final response boundary (or end-of-response event) before sending to TTS. If you must stream, only forward cleaned segments. • Filter/transform patterns: strip or replace known markers (e.g., 
<tool_call>
, bracketed text, explicit “thinking” phrases) or run a small post-process step to enforce an allowed output format. • Whitelist or classifier: enforce simple rules (must start with a sentence, no angle‑bracket tags, length limits) or run a short LLM/text classifier to rephrase/remove thought traces before TTS. • Centralized 
llm_node
processing: implement a node that inspects chunks, removes internal tags, and only yields cleaned text to the TTS node. When Live (Gemini Live) differs • Gemini Live provides explicit thinking metadata/events you can disable or ignore (via 
thinking_config
) so thinking is easier to separate from output. That option is not available for typical non‑live inference. If you want, paste a short raw snippet of the stream you’re seeing (a few tokens/chunks that include the thinking text and the intended response) and I’ll produce a precise 
llm_node
filter or regex to remove those tokens before TTS. Sources: LLM Output Replacement | LiveKit docs | Modifying LLM output before TTS | LiveKit docs | Large language models (LLM) | LiveKit Docs | Pipeline nodes and hooks | LiveKit docs | Gemini Live API plugin | LiveKit docs
6 Views
Open in Slack
PreviousNext
Powered by