Hi, I'm using SSML tags in my LiveKit agent and it works fine for speech, but the tags are showing up in the live transcription on screen. I've tried overriding

transcription_node

like this:

async def transcription_node(self, text_stream: AsyncIterable[str], model_settings: ModelSettings):
    async for text in super().transcription_node(text_stream, model_settings):
        if self.filter_ssml:
            text = self.ssml_tag_pattern.sub("", text)
        yield text

This works for messages sent using session.say(), but responses generated by the LLM are still not filtered. Does anyone know a way to clean or remove the SSML tags from live transcription for LLM outputs as well? Thanks a lot!

Thanks for reaching out, Gugi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I'm using a prompt template for the assistant/agent. Here's an example of the prompt:

PROMPT = """Your objective is to guide users in choosing the Wedding Venue and Date.
- Always detect and respond in {language}, except for numbers.
- Always follow this sequence:
    1. Ask for Venue and Date.
        - Say: "{coverage}"
        Note: 
        - Always use the current date (2025) and time to calculate the wedding date if the user gives a relative time.
        - If the user inputs a date without specifying the year, automatically use the current year.
        - If the user spells a venue name differently from the available options, automatically suggest the closest match."""

Then, I assign this prompt to my agent like this:

class WeddingBook(BaseAgent):
    def _init_(self, chat_ctx: ChatContext):
        super()._init_(
            instruction=PROMPT,
            chat_ctx=chat_ctx,
            template_name="wedding",
            next_agent=None
        )

For the message template, I use something like:

"What date and venue do you have in mind for your wedding? <break time=\"1.0s\" /> The package covers up to 250 guests, and any extras would be at your own cost."

This setup ensures the agent follows the wedding booking flow, asks the right questions, and provides consistent responses according to the template.

To help you remove SSML tags from LLM-generated responses in live transcription, you can gently intercept and clean the text before it reaches the TTS engine. Let me walk you through two friendly approaches you might find useful: 1. Using `before_tts_cb`: You can create a callback function that tidies up the text output from the LLM. Here’s an example to guide you:

async def _before_tts_cb(agent, text: str | AsyncIterable[str]):
    if isinstance(text, str):
        # Handle non-streaming text
        result = remove_ssml_tags(text)
        return result
    else:
        # Handle streaming text
        async def process_stream():
            async for chunk in text:
                processed = remove_ssml_tags(chunk)
                yield processed
        return process_stream()

Then, simply pass this callback when you initialize your agent:

agent = VoicePipelineAgent(
    vad=vad,
    stt=stt,
    llm=llm,
    tts=tts,
    before_tts_cb=_before_tts_cb,
)

2. Overriding `llm_node`: If you want a bit more control, you can override the llm_node method to clean the LLM’s text output directly:

async def llm_node(self, chat_ctx, tools, model_settings=None):
    async def process_stream():
        async with self.llm.chat(chat_ctx=chat_ctx, tools=tools) as stream:
            async for chunk in stream:
                if chunk is None:
                    continue
                content = getattr(chunk.delta, 'content', None) if hasattr(chunk, 'delta') else str(chunk)
                if content:
                    cleaned_content = remove_ssml_tags(content)
                    if cleaned_content != content and hasattr(chunk, 'delta') and hasattr(chunk.delta, 'content'):
                        chunk.delta.content = cleaned_content
                yield chunk
    return process_stream()

Helper Function: To make things easier, here’s a handy regex-based helper function that removes SSML tags:

import re

def remove_ssml_tags(text: str) -> str:
    ssml_pattern = re.compile(r"<[^>]+>")
    return ssml_pattern.sub("", text)

By using these approaches, you’ll make sure those SSML tags are gently stripped away from both streaming and non-streaming LLM outputs before they appear in your live transcription. If you have any questions or need further help, feel free to ask—I’m here to support you! Sources: chain-of-thought-tts/README.mdoc | Modifying LLM output before TTS | LiveKit Docs | pipeline-llm/llm_powered_content_filter.py | livekit-agents/livekit/agents/voice/transcription/filters.py | Amazon Polly TTS integration guide | LiveKit Docs

If the SSML tags are removed before the text reaches the TTS, what happens to the generated speech? Does that mean the speech will be produced without applying the SSML effects?

If SSML tags are removed before the text reaches the TTS engine, the generated speech won’t include the wonderful effects those tags add, like pauses, emphasis, or pitch changes. To keep those SSML effects in the speech while removing the tags from the live transcription, a great approach is to adjust the transcription pipeline instead of cleaning the text before it goes to TTS. Here’s a friendly way to handle it: 1. Intercept the transcription output and apply a cleaning function to gently remove SSML tags before showing it. 2. Make sure the original text with SSML tags stays intact and is sent to the TTS engine for speech synthesis. If you'd like, I’d be happy to walk you through how to set this up step-by-step. Just let me know!

Can you provide the code how to doing it?

To retain SSML effects in speech while removing the tags from live transcription, you can process the text separately for transcription and TTS. Below is a friendly example implementation to guide you through: 1. Define a helper function to remove SSML tags:

import re

def remove_ssml_tags(text: str) -> str:
    ssml_pattern = re.compile(r"<[^>]+>")
    return ssml_pattern.sub("", text)

2. Modify the transcription pipeline: Here, you’ll override the transcription method to clean the text for transcription while keeping the original text intact for TTS:

async def transcription_node(self, text_stream: AsyncIterable[str], model_settings: ModelSettings):
    async for text in super().transcription_node(text_stream, model_settings):
        # Clean text for transcription
        cleaned_text = remove_ssml_tags(text)
        yield cleaned_text

3. Pass the original text to TTS: Make sure the original text with SSML tags is sent to the TTS engine so it can produce speech with all the lovely effects. For example, if you’re using a callback:

async def _before_tts_cb(agent, text: str | AsyncIterable[str]):
    # Keep the text as is here to preserve SSML effects in speech
    return text

4. Integrate into your agent: Just ensure your transcription and TTS pipelines are set up to handle the cleaned and original text separately. This way, you’ll get: - Clean live transcription without any SSML tags. - Beautiful speech synthesis with all the intended SSML effects. If anything feels unclear or you’d like a hand with implementation, please don’t hesitate to ask—I’m here to help!