Hey, I'm once again asking for your support. I add...
# agents
f
Hey, I'm once again asking for your support. I added a thinking sound to my agent when it's doing function calls like checking calendars. Based on this example https://github.com/livekit/agents/blob/main/examples/voice_agents/background_audio.py everything works great for realtime models. However when using the setup for my STT - LLM - TTS pipeline my agent plays the thinking sound on almost every agent turn. So my guess is that the pipeline is going into "thinking" state when calling the llm (or stt / tts?). Is there a solution to this or should i remove the thinking audio and use the
background_audio.play()
method directly?
Copy code
background_audio = BackgroundAudioPlayer(
                thinking_sound=AudioConfig(
                    source=thinking_audio_path,
                    volume=0.6,
                ),
            )
            await background_audio.start(room=ctx.room, agent_session=session)
I liked the thinking_sound as it's way less work than manually setting this for function calls..
g
In the following example https://github.com/livekit/agents/blob/7757b0ef271706d9ff69c7780b06738b8e5558f0/examples/voice_agents/silent_function_call.py#L62 We have added an event listener to @session.on("function_tools_executed"). You can try adding a similar listener to play background audio, when ever a function is executed, instead of adding it on every function separately. Let me know if this works
f
I’ll check it out, thanks! But this might interfere with the agent talking. I think instead of a workaround it would make sense to specify the thinking state of the pipeline agent differently. So the usual llm calls shouldn’t make the agent “think” as this is the normal agent turn
g
I believe it was considered thinking because, if we mimic a human conversation with voice agents, the llm are the brain, so whenever llm is doing something that will be thinking.
f
But then basically every agent turn is a thinking one. Or can you somehow specify the threshold for the latency? Like only play the thinking music after 500ms or so. Because latency with pipeline is 200-500ms
r
The thinking sound will be played while the LLM is "thinking". Is that not what you want?
c
I am not sure the sound is stopped when the agent starts speaking. I added a sound for agent state "thinking" but it keeps playing when the agent is speaking until the penultimate few tokens of speech, where only agent speech is heard. Can we stop the sound when the agent starts it's speech?
f
@refined-appointment-81829 okay but when exactly is the LLM "thinking" . In my pipeline agent i have a background audio music file present. So an interaction looks like this:
user: "hello"
<music> // unexpected
agent: "hello, how are you?"
user: "i'm fine thanks. When's david available? I need an appointment"
<music> // unexpected
agent: "great! let me check his calendar, this might take a moment!
<agent_tool_call>
<music> // EXPECTED
....
agent: "alright, david is available starting at 3pm until 5pm today..."
With realtime models this works, but i think for pipeline agents every llm call is interpreted as "thinking" state which in a normal conversation would result in the music being played every time the agent answers. I can record a short voice sample so you can hear what i mean. Hence my suggestion here:
I think instead of a workaround it would make sense to specify the thinking state of the pipeline agent differently. So the usual llm calls shouldn’t make the agent “think” as this is the normal agent turn
or
can you somehow specify the threshold for the latency? Like only play the thinking music after 500ms or so. Because latency with pipeline is 200-500ms
r
> When exactly is the LLM thinking: Have a look at this code just search for
thinking
https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/voice/agent_activity.py Quickly put this diagram together just not as a basic outline
f
Hi @few-zebra-58968 can you provide this thinking clip to me too? It will be really helpful of you!