Problem
My agent is a voice assistant, but many of...
# ask-ais
square-activity-46232
03/09/2026, 6:21 PMProblem
My agent is a voice assistant, but many of the tools it uses return structured responses that include things like:
• URLs (Google Meet links, Maps links, etc.)
• Email addresses
• Metadata
When the LLM reads those links out loud, the user experience becomes very bad. For example, the agent might start pronouncing something like:
"HTTPS colon slash slash meet dot google dot com slash..."
That completely breaks the voice interaction.
To mitigate this, I added rules in the system prompt like:
• Do not respond with links
• Convert structured tool responses into natural spoken language
• Do not read JSON, logs, or technical structures
• Convert email addresses to spoken format
• Express dates and times naturally
This works well for voice output.
However, the product team now wants the user to still see the links on screen, even if the agent does not say them aloud.
Goal
We want:
1. The agent to speak a natural response
2. The link to appear in the UI
3. The link to be stored in the conversation context
4. The agent to NOT pronounce the link
Example
User:
"Create a meeting with Carlos tomorrow at 7pm."
Flow:
1. Agent:
"Give me a second while I create the meeting."
2. Agent calls Calendar tool
3. Tool returns:
Copy code
{
"meet_link": "<https://meet.google.com/abc-defg-hij>"
}send_visual_message(content: string)t
tall-airline-16756
03/09/2026, 6:21 PMThanks for reaching out, Roman. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
tall-airline-16756
03/09/2026, 6:22 PMThanks — this is a great, well-scoped problem and your proposed tool is a sensible starting point. I’ll respond to the three questions you asked and give a concise, idiomatic LiveKit design you can implement that meets all four of your requirements.
Quick acknowledgement of what I’m answering
• You want the assistant to speak a natural sentence, show the link in the UI, persist the link in conversation history, and ensure the assistant does not speak the link aloud.
Short answer / recommendation
• Use a two-step approach:
a. Let the agent produce its natural spoken reply (speech output).
b. Publish the structured content (link, emails, metadata) as a visual-only assistant message by sending it into LiveKit’s text/transcription stream and also injecting it into the chat context.
This yields voice-safe speech, visible UI content, and persistent history the LLM can read on the next turn.
Why this is better than a single “no-voice” assistant mode
• It cleanly separates concerns: audio playout vs. UI/display vs. memory.
• The frontend can render visual-only messages differently (e.g., clickable links) while ignoring them for TTS.
• The LLM sees the link as part of assistant history, so future turns have consistent context.
Concrete design pattern (minimal, idiomatic)
1) Agent creates the meeting and receives structured tool output:
• e.g. { "meet_link": "https://meet.google.com/abc-defg-hij" }
2) Agent speaks the natural response:
• e.g. call your speech API or LiveKit voice function to say: “Your meeting with Carlos for tomorrow at 7pm has been created.”
3) Agent publishes a visual-only assistant message (tool like your proposed send_visual_message):
• Publish the URL to LiveKit’s text/transcription topic (visual-only), not as audio.
• Immediately add the same content as an assistant message to the conversation context / chat history so the LLM can read it later.
Why both publishing and context injection are needed
• Publishing to the text/transcription stream makes the link visible to the user and lets the UI render a clickable link (but not play it as audio).
• Adding it to the chat context ensures the conversation history reflects what the assistant “sent” so the LLM can reference it later.
Example flow (pseudocode)
1. Agent does the action and receives the link.
2. Agent: session.say("Your meeting with Carlos has been created.") // spoken
3. Agent: send_visual_message("https://meet.google.com/abc-defg-hij")
◦ Implementation of send_visual_message should:
▪︎ publish the string to the transcription/text stream (visual only)
▪︎ add an assistant chat message with that string to the persistent chat context
Notes on implementation details (practical tips)
• Mark the published visual message as visual-only (metadata/attribute) so the frontend and any voice pipeline can skip it for TTS.
• Keep the visual message content simple (the raw link or a short label + link). If you need human-friendly presentation, send both a short spoken phrase and the raw link in the visual message.
• If you need richer structured data (emails, times), you can include both a human-readable summary in the spoken message and structured JSON in the visual message metadata — but make sure the speech output never includes raw URLs or JSON.
• Make the send_visual_message tool idempotent or return confirmation so agent logic can handle retries gracefully.
Answers to your explicit questions
1. Is this the right architectural approach, or is there a more idiomatic LiveKit pattern?
◦ Yes, the two-step approach (speech + visual-only text stream + context injection) is the idiomatic LiveKit pattern. It separates audio from UI while preserving context and is preferable to inventing a completely separate “assistant message with voice disabled” mode.
1. Is there a recommended way to separate voice output from assistant messages?
◦ Yes: use the standard speech/tts for spoken output, and publish visual content to the text/transcription stream (or equivalent text channel) while adding the same content to the assistant chat history. The frontend should render the transcription/text messages visually and not route them into TTS.
1. Has anyone implemented voice-safe tool responses vs visual messages?
◦ Yes — patterns where structured data or attachments are published as non-spoken visual messages and the agent speaks a human-friendly summary are used. The key pieces are: (a) visual-only publishing, (b) marking metadata so TTS skips it, and (c) injecting into chat memory.
Minimal example tool (logic you can adapt)
• send_visual_message(content):
◦ publish content to the text/transcription topic (visual only)
◦ add an assistant message with content to the chat context
◦ return no spoken output
UX examples
• Spoken: “Your meeting with Carlos for tomorrow at 7pm has been created.”
• Visual assistant message (below speech bubble or in attachments): clickable link https://meet.google.com/abc-defg-hij
• Later, LLM can reference that URL from the assistant message in history.
Potential edge cases to plan for
• If you redact links visually for security, the LLM will still need the original link in context; consider restricted visibility flags if necessary.
• If a user asks the assistant to “read the link,” define explicit behaviors (e.g., speak a short label, not the raw URL).
• For transcripts shown to users with TTS playback enabled, ensure the frontend excludes visual-only messages from TTS activation.
If you’d like, I can:
• Draft the exact send_visual_message function in your stack (JS/TS or Python).
• Provide sample frontend handling rules for rendering visual-only messages and excluding them from TTS.
• Show an example conversation with the chat context entries you’d persist.
Which would help most next — a code sketch for your backend tool or the UI rendering rules?
Sources: Prompting guide | LiveKit Documentation | How to build an agent with speech-to-text input and text-only output | Live... | Pipeline nodes and hooks | LiveKit Documentation | Text and transcriptions | LiveKit Documentation | Playing Audio | LiveKit Documentation
s
square-activity-46232
03/09/2026, 6:27 PMdoes this need a change in the client side to be able to receive and show the topic in the UI or is it done automatically?
t
tall-airline-16756
03/09/2026, 6:27 PMShort answer: Yes — you need client-side changes. It’s not automatic.
Why: Visual-only assistant messages are published to the text/transcription stream (e.g., topic
lk.transcriptionregisterTextStreamHandler('lk.transcription', ...)useTranscriptions()lk.transcribed_track_id Copy code
room.registerTextStreamHandler('lk.transcription', async (reader, participantInfo) => {
const text = await reader.readAll();
const isAudioTranscription = !!reader.info.attributes['lk.transcribed_track_id'];
if (isAudioTranscription) {
// show as live transcription
} else {
// show as visual-only assistant message (do NOT send to TTS)
}
});s
square-activity-46232
03/09/2026, 6:30 PMi need a solution that can be managed only in the server (livekit app)
t
tall-airline-16756
03/09/2026, 6:30 PMThanks — I understand you want a server-only solution that requires no client changes. Short answer: you cannot fully guarantee the UI will show visual-only assistant messages without at least minimal client-side support. I’ll explain why, then give the best server-side approach that minimizes client work and exactly what small client behavior is required.
Why a purely server-only solution is not possible
• The server can publish visual-only messages into LiveKit’s text/transcription stream and inject assistant messages into the conversation history, but the client must still be able to receive and render those text-stream messages (or read the chat history). Built-in playgrounds or default clients do not automatically display text-stream messages as visual assistant messages in all cases.
• The server cannot force a client to render a text stream as a clickable link or to skip TTS playback — the client controls rendering and audio playback.
What you can do entirely on the server (maximal server-side work)
• Do everything server-side except the final rendering/TTS decision:
a. Create the resource (calendar event) and obtain the link.
b. Produce the spoken response via your agent (server triggers TTS / speech playback to the room).
c. Publish the link to the text/transcription topic (e.g., topic "lk.transcription") so it reaches clients that subscribe to it.
d. Insert the same link as an assistant message into the persistent chat context/history so future LLM turns can read it.
Why this minimizes client work
• All logic, formatting, and persistence live on the server. The client only needs to display whatever it already consumes from the room text stream or the persisted chat history.
• If the client already subscribes to transcription/text streams or reads assistant chat history, no change is necessary on clients that already do that.
Minimal client-side requirement (unavoidable)
• The client must either:
◦ Subscribe to the text/transcription stream (topic) and render those messages in the chat UI, OR
◦ Read assistant chat history from the persisted conversation context and render those messages.
• Additionally, the client must ensure visual-only messages are not fed into TTS. That can be done by either:
◦ Recognizing that the message came from the text stream and has no audio track attribute, or
◦ Reading metadata/attributes you attach (for example, is_visual_only=true) and skipping TTS.
Two pragmatic options depending on how much you can change clients
Option A — Server-heavy, client with existing transcription support (minimal/no client change)
• Server publishes visual-only message to text stream and injects into chat context.
• If client already uses transcription handlers or a useTranscriptions-style hook, it will display the link automatically and you won’t need to change code.
Option B — Server-heavy, client needs a one-line update (recommended minimal change)
• Server does the same publishing + context injection and adds a small attribute flag (e.g., is_visual_only=true) when sending the text stream.
• Client only needs a single small change: when receiving text stream messages, check that attribute and:
◦ Render message as visual card (link),
◦ Do not send to TTS.
This is a one-time, tiny client change and keeps all business logic on the server.
Implementation sketch (server responsibilities)
1. After creating the meeting:
◦ Call TTS/speech send for: “Your meeting with Carlos for tomorrow at 7pm has been created.”
1. Publish visual-only text to transcription:
◦ topic: lk.transcription
◦ text: https://meet.google.com/abc-defg-hij
◦ attributes: { is_visual_only: "true", lk_transcribed_track_id: "" }
1. Add assistant message to conversation history:
◦ role: assistant
◦ content: "https://meet.google.com/abc-defg-hij"
◦ optional metadata: { visual_only: true }
1. Optionally return an API/ack to the client so the client can refresh messages if it does not subscribe to the stream.
What I recommend
• If you control most clients and can ship a tiny client change, go with Option B. It keeps all logic on the server but makes rendering/TTS decisions explicit and robust.
• If you cannot change any client and some clients already subscribe to text/transcription streams or read chat history, the server-only approach may work for those clients — but you cannot guarantee it for all clients universally.
If you want, I can:
• Draft the exact server-side code snippet for publishing to the transcription stream and injecting into chat context in your stack (Python or Node).
• Draft the minimal client snippet (one small conditional) to ignore visual-only messages for TTS and render them as links.
Which snippet would you prefer first — server (Python/Node) or the minimal client change?
s
square-activity-46232
03/09/2026, 6:32 PMbut it's very easy, I turn off tts (voice_enabled_false), use the reply method, and then turn it on again,
the result is: the agent replies to the client but without reading aloud.
t
tall-airline-16756
03/09/2026, 6:33 PMYou're on the right track — toggling TTS off, sending the reply, and toggling it back on is a valid, practical server-side pattern to produce visual-only assistant messages while keeping spoken responses natural. Below is a concise confirmation plus best-practice notes and small caveats to make it robust.
What your approach achieves
• The assistant sends the text reply to clients (so the link appears in UI).
• No audio track / TTS is published for that reply (agent does not read the link aloud).
• You can re-enable TTS afterwards for subsequent spoken replies.
• The message can be persisted to conversation history so the LLM can access it later.
Recommended implementation (server-side)
1. Disable audio output for the session or output channel before sending the visual-only message:
◦ e.g., set
audio_output=falsesession.output.set_audio_enabled(False)is_visual_only=truesend_visual_messageis_visual_only3 Views