https://livekit.io logo
Join Slack
Powered by
how can i remove ssml tags from transcription? i d...
# ask-ai
b
how can i remove ssml tags from transcription? i don’t want it to be visible in the UI or in chat context. only needed for TTS plugin
t
Thanks for reaching out, Nika. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I understand you want SSML tags available to the TTS plugin but hidden from the UI and chat history. You can do that by preserving SSML in the LLM/TTS path while stripping tags from the transcription path before it’s saved or sent to the frontend. Below is a concise approach and an example you can adapt. What I’m addressing • Keeping SSML for TTS output. • Removing SSML from text shown in the UI or stored in chat context. Solution (high-level) • Let the LLM output include SSML so the TTS plugin receives it unchanged. • Intercept transcription/chat output and remove SSML tags before writing to chat history or sending to the frontend. • Implement this by customizing the agent’s 
llm_node
(pass-through for TTS) and 
transcription_node
(strip tags for UI). Recommended implementation (Python example) 1. Use a regex to remove XML/SSML tags when streaming text to the UI/chat. 2. Preserve the raw LLM stream for the TTS plugin.
Copy code
import re
from livekit.agents import Agent
from typing import AsyncIterable

class MyAgent(Agent):
    async def llm_node(self, chat_ctx, tools, model_settings=None):
        # Pass raw LLM output (with SSML) through so TTS can use it
        async with self.llm.chat(chat_ctx=chat_ctx, tools=tools) as stream:
            async for chunk in stream:
                yield chunk

    async def transcription_node(self, text: AsyncIterable[str], model_settings=None) -> AsyncIterable[str]:
        # Regex to remove SSML/XML tags like <prosody>, <break>, etc.
        ssml_regex = re.compile(r'<[^>]+>')
        async for chunk in text:
            clean_text = ssml_regex.sub('', chunk)
            yield clean_text
Behavior summary • TTS receives the original text with SSML (so prosody, pauses, etc. work). • UI and chat history receive cleaned text without SSML tags. Notes and tips • The regex above strips all XML-like tags. If you need to preserve specific tags or attributes, adjust the regex or implement an XML parser to remove only SSML namespaces/tags. • Test edge cases where the LLM might generate malformed tags; consider using an XML parser (like Python’s xml.etree) for stricter handling if needed. • This pattern is intended for agent pipelines that support 
llm_node
and 
transcription_node
. If you share which agent architecture or language/runtime you’re using (or an example LLM/TTS stack), I can give a tailored snippet or a parser-based variant that preserves allowed tags and removes everything else. Sources: Modifying LLM output before TTS | LiveKit Documentation | LLM Output Replacement | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation | How to build an agent with speech-to-text input and text-only output | Live...
b
yes but what if i have streaming transcription ? SSML still visible until it has close tag, then it’s removed
t
Thanks — great follow-up. You’re right: with streaming transcription a naive tag-stripper can still show partial SSML (e.g., an opening tag) until the closing tag appears. Below are practical patterns you can use so SSML never appears in the UI while still letting TTS receive the full SSML. What I’m addressing • Prevent any SSML fragments appearing in the UI during streaming. • Preserve full SSML for TTS so speech synthesis still benefits from tags. Options you can use (pick one based on latency vs. complexity) • Buffer-and-release (simple, safest) ◦ Buffer incoming transcription chunks until a complete tag-balanced segment is available, then emit the cleaned text. ◦ Behavior: UI shows only fully-cleaned sentences/segments; small extra delay while waiting for matching closing tags. ◦ When to use: When a little display latency is acceptable and you want the simplest reliable solution. • Incremental filtering with tag-state machine (low-latency, moderate complexity) ◦ Maintain a small parser state while streaming that tracks whether you are inside an open SSML tag. ◦ For each incoming chunk: i. Append to a short buffer. ii. Remove any complete tags (e.g., <...>), but if you detect an unclosed '<' at the chunk end, keep that partial content out of the emitted UI text until the close appears. iii. Emit only text outside of tags and any fully-closed tag regions removed. ◦ Behavior: Low latency; partial plain text appears immediately; no tag fragments are emitted because any open-tag portion is withheld until it closes (or timed out). ◦ When to use: When you need near-real-time UI updates but must avoid showing tag fragments. • Use an XML/SSML incremental parser library (robust, recommended for edge cases) ◦ Use a streaming XML parser (SAX-like) that can parse partial input and inform you when character data is safe to emit. ◦ Feed chunks into the parser; it will call character-data handlers only for text nodes (not for tags). For partial tags, the parser will not emit until the tag is complete. ◦ Behavior: Most robust for malformed or nested tags; avoids regex pitfalls. ◦ When to use: Production systems or when LLM may produce malformed SSML. • Emit cleaned UI text but keep full SSML for TTS in parallel ◦ Always build two streams: ▪︎ UI stream: The cleaned text produced by one of the methods above. ▪︎ TTS stream: The raw LLM output (including SSML) forwarded to the TTS plugin. ◦ Ensure TTS receives the original stream immediately while UI uses the filtered stream. Implementation guidance and snippets 1) Minimal stateful approach (pseudocode outline) • Keep a small string buffer and boolean in_tag = False. • For each incoming chunk: ◦ For each char in chunk: ▪︎ If char == '<': set in_tag = True; start skipping chars. ▪︎ If char == '>' and in_tag: set in_tag = False; discard tag content. ▪︎ If not in_tag: append char to emit buffer. ◦ Emit any newly accumulated characters that are outside tags. • Add a safety timeout: if a tag remains unclosed beyond X ms or bytes, decide to drop the incomplete tag and emit the buffered text anyway. 2) Regex is not sufficient for streaming because it can’t safely handle partial tags at chunk boundaries. Prefer state machine or streaming XML parser. 3) Example library choices • For Python: use a SAX/expat incremental parser (xml.parsers.expat) and feed bytes as they arrive. • For Node.js: use sax-js or an incremental XML parser. • For browser JS: use a streaming parser or implement the lightweight state machine above. Edge cases & recommendations • Malformed tags: use an incremental parser or include a fallback that times out partial tags to avoid indefinite blocking. • Latency vs completeness: Buffering whole sentences prevents fragments but increases latency. The incremental state machine minimizes latency while preventing tag fragments. • Testing: Simulate chunk boundaries that split tags (e.g., "<pros" then "ody rate='fast'>") to ensure your logic never emits "<pros" to the UI. If you tell me which runtime/language you’re using (Python, Node, browser JS) and your latency tolerance (acceptable delay vs. immediate updates), I’ll provide a concrete code example you can drop into your transcription pipeline (including a timeout-safe variant).
5 Views
Open in Slack
PreviousNext
Powered by