Amazon Nova Pro speaks chain-of-thought tokens in pipeline mode

Thanks for reaching out, YASH. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I am observing the model outputting internal chain-of-thought-style tokens (step-by-step reasoning) in its responses and want to stop it. This is agent configuration

stt = deepgram.STT(model="nova-2")
        llm = aws.llm.LLM(
            model="us.amazon.nova-2-lite-v1:0",
            region=os.getenv("AWS_REGION", "us-east-1"),
        )
        tts = aws.tts.TTS(voice="Matthew", speech_engine="generative", sample_rate=24000, region=os.getenv("AWS_REGION", "us-east-1"))
        # attach_api_call_logging(stt=stt, llm=llm, tts=tts)
        session = AgentSession[ConversationState](
            userdata=userdata,
            stt=stt,
            llm=llm,
            tts=tts,
            turn_detection=MultilingualModel(),
            preemptive_generation=True,
            vad=silero.VAD.load(
                min_speech_duration=0.02,
                min_silence_duration=0.3,
                prefix_padding_duration=0.2,
                sample_rate=16000,
            ),
        )

Thanks — that’s clear and helpful. You’re seeing the Nova model verbalize its internal step-by-step reasoning in pipeline mode and you want that stopped. I’ll walk through why that happens and give practical, low-risk fixes you can apply to your current agent setup. Acknowledgement • You’re using STT → LLM → TTS pipeline with

us.amazon.nova-2-lite-v1:0

as the LLM. • The LLM output is being sent to TTS, so any chain-of-thought text the model emits gets spoken. Why this happens • The model can include internal reasoning or step-by-step traces in its output when the prompt allows or doesn’t explicitly forbid it. • In a pipeline (non-realtime) flow the model’s full textual output is passed to TTS unless you remove or alter it first. What you can do now (practical fixes) 1) Strong prompt instruction to suppress reasoning (lowest effort) • Add a short, explicit instruction in your agent/system prompt telling the model not to show internal thoughts and to return only the final answer. • Example instruction text to include in your agent configuration: ◦ “Do not show your chain-of-thought or internal reasoning. Only produce the final answer that should be spoken.” • This is usually effective and safe to try first. 2) Enforce an output format (structured final-only response) • Ask the model to return only a specific field that contains the spoken text, e.g.: ◦ “Respond only with JSON: { "speak": "<final reply>" }. Do not include any other text or reasoning.” • Then parse the LLM output in your pipeline and pass only the

speak

value to TTS. 3) Post-process/filter LLM output before TTS (robust) • Intercept the LLM output node and strip suspected reasoning tokens or sections before calling TTS. • Simple strategy: ◦ If you see markers like “Thought:” or step numbers, remove everything from that marker onward. ◦ If you require stronger guarantees, parse for JSON or a delimiter and only use the intended field. • This is the most reliable defensive approach because it does not rely solely on prompt compliance. 4) Combine prompt + filtering (recommended) • Use an explicit suppression prompt AND a post-processing filter so that even if the model slips, TTS never speaks internal steps. Concrete example you can plug into your existing code 1. Update the system/instructions when creating the agent or when invoking the LLM:

instructions = """
You are a concise voice assistant.
Only produce the final reply to be spoken. Do NOT show internal thoughts, chain-of-thought, step-by-step reasoning, or debugging traces.
If asked to reason, provide only the final answer.
If possible, return output as JSON: {"speak": "<final reply>"}.
"""

# Pass instructions to your agent / LLM call as applicable.

1. Filter output before TTS (pseudocode)

response = llm.generate(prompt=combined_prompt, ...)
text = response.text

# If the model returns JSON, parse and take the "speak" field.
try:
    speak = json.loads(text).get("speak")
except:
    # Fallback: strip any 'Thought:' or numbered reasoning
    speak = re.split(r"\bThoughts?:|\bThought:|1\.", text, maxsplit=1)[0].strip()

tts.speak(speak)

Questions to help me refine the recommendation • Do you already add a system/instructions string to the LLM call or Agent initialization? If so, can you paste it (redacted)? • Do you prefer a prompt-only fix or do you want me to show a small code example that intercepts and filters LLM output in your current pipeline? • Do you see any consistent markers in the outputs that indicate reasoning (e.g., “Thought:”, “Step 1:”, numbered lists)? If you want, share a redacted example of an input and the LLM output that got spoken (or a log snippet). I’ll propose the exact prompt text and a short filter that matches that output so TTS never speaks the reasoning. Sources: Amazon Bedrock LLM plugin guide | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation

Sorry this is my code

stt = deepgram.STT(model="nova-2")
        llm = aws.llm.LLM(
            model="us.amazon.nova-pro-v1:0",
            region=os.getenv("AWS_REGION", "us-east-1"),
        )
        tts = aws.tts.TTS(voice="Matthew", speech_engine="generative", sample_rate=24000, region=os.getenv("AWS_REGION", "us-east-1"))
        # attach_api_call_logging(stt=stt, llm=llm, tts=tts)
        session = AgentSession[ConversationState](
            userdata=userdata,
            stt=stt,
            llm=llm,
            tts=tts,
            turn_detection=MultilingualModel(),
            preemptive_generation=True,
            vad=silero.VAD.load(
                min_speech_duration=0.02,
                min_silence_duration=0.3,
                prefix_padding_duration=0.2,
                sample_rate=16000,
            ),
        )

Thanks — this is very helpful. I see you’re using

us.amazon.nova-pro-v1:0

in an STT → LLM → TTS pipeline and the model is verbalizing its chain-of-thought. I’ll give a concise, actionable fix you can drop into your project: reinforce the prompt and intercept/filter the LLM stream before it goes to TTS. What I’m responding to • You want to stop internal chain-of-thought or reasoning traces from being spoken when Nova Pro is used in pipeline mode. Two minimal, complementary fixes (use both for best reliability) 1) Strong instruction/prompt (easy, low-risk) • Add an explicit instruction so the model returns only the final answer to be spoken:

instructions = """
You are a concise voice assistant. Only output the final reply to be spoken.
Do NOT reveal your internal thoughts, step-by-step reasoning, or any debugging traces.
If possible, return output as JSON: {"speak": "<final reply>"} and nothing else.
"""
# Pass this instructions string into your Agent or LLM call.

• This reduces the chance the model will generate reasoning text, but prompts alone aren’t 100% guaranteed. 2) Filter the LLM output stream before handing it to TTS (reliable) • Intercept the streaming chunks from the LLM and sanitize them (remove known reasoning markers or only accept a specific JSON field). • Implement a custom Agent that overrides

llm_node

to inspect and clean

chunk.delta.content

before yielding to downstream nodes. Concrete example sketch (adapt to your imports and runtime) 1) Custom Agent with

llm_node

override:

import re
import json
from livekit.agents import Agent

class FilteredNovaAgent(Agent):
    def __init__(self):
        super().__init__(instructions=instructions)

    async def llm_node(self, chat_ctx, tools, model_settings=None):
        async def process_stream():
            async with self.session.llm.chat(chat_ctx=chat_ctx, tools=tools, tool_choice=None) as stream:
                async for chunk in stream:
                    if chunk is None:
                        continue

                    # Extract delta text safely
                    content = getattr(getattr(chunk, "delta", None), "content", None)
                    if content is None:
                        yield chunk
                        continue

                    # 1) Try to parse JSON with {"speak": "..."} and prefer that
                    speak_text = None
                    try:
                        parsed = json.loads(content)
                        if isinstance(parsed, dict) and "speak" in parsed:
                            speak_text = parsed["speak"]
                    except Exception:
                        pass

                    # 2) If no JSON, strip common reasoning markers
                    if speak_text is None:
                        # Remove blocks starting with markers like "Thought:", "Thoughts:", "Step 1:", or tags like "<tool_call>"
                        cleaned = re.split(r"\bThoughts?:|\bThought:|<tool_call>|(^\d+\.)", content, maxsplit=1)[0].strip()
                        speak_text = cleaned

                    # Replace the chunk content with sanitized speak_text
                    if hasattr(chunk, "delta") and hasattr(chunk.delta, "content"):
                        chunk.delta.content = speak_text

                    yield chunk

        return process_stream()

2) Use the agent in your session (adjust imports to match your project):

# models (your existing setup)
stt = deepgram.STT(model="nova-2")
llm = aws.LLM(model="us.amazon.nova-pro-v1:0", region=os.getenv("AWS_REGION","us-east-1"))
tts = aws.TTS(voice="Matthew", speech_engine="generative", sample_rate=24000, region=os.getenv("AWS_REGION","us-east-1"))

session = AgentSession(
    stt=stt,
    llm=llm,
    tts=tts,
    turn_detection=MultilingualModel(),
    preemptive_generation=True,
    vad=silero.VAD.load(...),
)

agent = FilteredNovaAgent()
await session.start(agent=agent, room=ctx.room)

Notes and small tips • Log raw chunks first to identify exact markers Nova Pro outputs (e.g., “Thought:”, numbered steps, or tags like

<tool_call>

). Then refine your regex or JSON parsing to match those tokens exactly. • Combining explicit prompt instructions and stream filtering gives the strongest guarantee TTS won’t speak internal reasoning. • If you control the code path that calls the LLM directly (non-streaming), you can apply the same filtering logic on the whole text response before TTS. Would you like: • A tailored regex to match the exact tokens you’re seeing if you paste one redacted example of the raw LLM output? • Or a complete runnable example adjusted to your exact imports so you can paste-and-run? I can provide either next — tell me which you prefer and, if possible, a short redacted sample of the problematic output. Sources: AWS AI and LiveKit | LiveKit Documentation | Simple Content Filter | LiveKit Documentation | LLM-Powered Content Filter | LiveKit Documentation | LLM Output Replacement | LiveKit Documentation