Hi, Is there a recommended way to mute the bot without consuming TTS API costs, while also not losing TTS transcription chunks if the audio is muted/unmuted while the agent is speaking?

You could hook the TTS node and not send it to TTS server during whatever state you want to do that for

@refined-appointment-81829 I do not think this is an option, because of this scenario: 1. Bot starts talking 2. User mutes bot -> TTS interrupted 3. User unmutes bot -> TTS still interrupted Expected behaviour: Bot continues speaking with step 3 -> either where he left of or where he would actually be if he would have not been muted.

What does "mute" mean in this case?

That would mean: 1. the frontend would definitely not play any TTS audio -> We currently just mute the audio element in the web frontend, though we wish to also not consume any TTS API credits, because maybe the user keeps the bot muted for most of the session and therefore we do not need to spent API credits for no reason. 2. and the backend possible does not send any TTS stream chunks via lk.transcription (though already processed TTS audio does not need to be "thrown away", in case bot directly gets unmuted again within the same reply).

So, my definition is this: • Agent has a flag to do TTS or Not (proxy for mute/unmute) • Client uses RPC to set/unset that flag • If flag is not set (unmuted) it is normal • If flag is set (muted) the logic in the the TTS node will block the TTS API calls but it will still do the transcript bit I don't think I've done this before but that is how I would approach the issue if I were trying to solve it. Another option is you could potentially make a

/dev/null

TTS as another way and switch between real TTS and mock TTS.

Yes, so the only way to save TTS credits (what we think) is if the bot is muted and the user prompts the agent afterwards. Then the TTS would have never been called and the user would also understand, that if he unmutes the bot right when it started generating a text output, that the agent does not start talking since he was muted during the prompting.

I am not sure... I expected that you will be paying TTS rather the user hears it or not. In most cases I think of Mute/Unmute on the client side as muting the user so that would be STT. But looks like you have a way to mute/unmute the agent track.

Yes, we actually just have a 1:1 chat session between a human and a bot, so this is unrelated. And yeah we actually do mute the agent track.

Yes, I think it should not be too hard. But not free of work either. Incase it helps I just dropped this thread into Devin and this is what it says about it: > Based on the conversation, you're asking about implementing a mute feature for the agent that avoids TTS API costs while preserving transcription chunks when the bot is muted/unmuted during speech. > > Current Architecture > > The LiveKit Agents framework processes speech through a pipeline where TTS generation and transcription forwarding happen in parallel. agent_activity.py:1772-1796 The

tts_node

is called to generate audio, and separately, the

transcription_node

processes text for transcription output. agent_activity.py:1800-1812 > > Recommended Approach > > The suggestion from CWilson to override the

tts_node

is the correct approach. You can customize the

tts_node

method in your

Agent

subclass to conditionally skip TTS API calls based on a mute flag: agent.py:314-339 > > Here's how this works: > 1. Override `tts_node`: Create a custom implementation that checks your mute state before calling TTS > 2. Return empty audio stream when muted: When muted, return an empty

AsyncIterable[rtc.AudioFrame]

or

None

to skip TTS generation agent.py:316-320 > 3. Transcription continues independently: The

transcription_node

runs separately from

tts_node

, so transcriptions will still be generated from the LLM text output agent_activity.py:1800-1812 > > Implementation Pattern >

class MutableAgent(Agent):  
>     def __init__(self, *args, **kwargs):  
>         super().__init__(*args, **kwargs)  
>         self.is_muted = False  
>       
>     async def tts_node(self, text: AsyncIterable[str], model_settings: ModelSettings):  
>         if self.is_muted:  
>             # Consume text but don't generate audio  
>             async def _empty_audio():  
>                 async for _ in text:  
>                     pass  # Consume text stream  
>                 return  
>             return _empty_audio()  
>           
>         # Normal TTS processing  
>         return await super().tts_node(text, model_settings)

> > Important Limitations > > You cannot resume interrupted TTS audio. Once TTS generation is interrupted (whether by user interruption or muting), the audio generation stops and cannot be resumed from where it left off. agent_activity.py:1823-1828 The framework clears the audio buffer on interruption. agent_activity.py:1826-1828 > If you mute mid-speech, the TTS that was already generated will be lost. When you unmute, you would need to trigger a new response (e.g., using

session.generate_reply()

) to get the agent to speak again. > > Notes > > The separation between audio generation (

tts_node

) and transcription (

transcription_node

) in the pipeline architecture makes this approach viable. agent_activity.py:1800-1812 Transcriptions are forwarded independently of audio, so muting TTS won't affect transcription delivery. However, the framework doesn't support pausing/resuming TTS generation - it's an all-or-nothing operation per speech turn. > > References: > • https://github.com/livekit/agents/blob/d3e4279e/livekit-agents/livekit/agents/voice/agent_activity.py#L1772-L1796 > • https://github.com/livekit/agents/blob/d3e4279e/livekit-agents/livekit/agents/voice/agent.py#L314-L339

🤔 Yes that is what we did, right @able-branch-17267? The thing is that it stopped sending transcripts when I unmute him (while he still responds). Meaning it was doing this for us: Bot starts talking -> user mutes bot and RPC is called -> text transcriptions continue being sent -> user unmutes the bot -> transcriptions are no longer sent. I think this is what we experienced.

couldn't you just conditionally yield?

# Insert custom text processing here
    async for frame in Agent.default.tts_node(self, text, model_settings):
        # just don't yield frame if mute state is true
        yield frame