https://livekit.io logo
Join Slack
Powered by
<@U084APK40NS> I want to block on the LLM saying s...
# ask-ai
a
@millions-hair-24956 I want to block on the LLM saying something, then I want to take an action. How do I do that?
g
To block on the LLM saying something and then take an action, you can use the 
before_llm_cb
callback in the VoicePipelineAgent. This callback allows you to intercept and modify the LLM's response before it is synthesized into speech. By returning 
False
from this callback, you can cancel the synthesis of the reply, effectively blocking the LLM from saying anything. You can then take your desired action before allowing the LLM to proceed. Here's a basic example of how you might set this up:
Copy code
from livekit.agents.pipeline import VoicePipelineAgent

def before_llm_callback(agent, chat_ctx):
    # Your custom logic here
    # For example, log the LLM's intended response
    print("LLM is about to say:", chat_ctx.messages[-1].text)
    
    # Take your action here
    # ...

    # Return False to block the LLM from speaking
    return False

agent = VoicePipelineAgent(
    vad=vad_instance,
    stt=stt_instance,
    llm=llm_instance,
    tts=tts_instance,
    before_llm_cb=before_llm_callback
)
Would you like more details on implementing this or any specific part of the process? Sources: - VoicePipelineAgent 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
a
@millions-hair-24956 no I want the bot to say something then hang up
g
To have the bot say something and then hang up, you can use the 
agent_speech_committed
event to trigger the hang-up action after the bot has spoken. Here's a basic example of how you might implement this: 1. Define a Callback: Use the 
agent_speech_committed
event to detect when the bot has finished speaking. 2. Hang Up: Call a function to hang up the call after the speech is committed. Here's a code snippet to illustrate this:
Copy code
import asyncio
from livekit import api
from livekit.agents.pipeline import VoicePipelineAgent

async def hang_up(ctx, participant):
    lk_api = api.LiveKitAPI()
    await lk_api.room.remove_participant(
        api.RoomParticipantIdentity(
            room=ctx.room.name, identity=participant.identity
        )
    )

def on_speech_committed(ctx, participant):
    asyncio.create_task(hang_up(ctx, participant))

agent = VoicePipelineAgent(
    vad=vad_instance,
    stt=stt_instance,
    llm=llm_instance,
    tts=tts_instance
)

agent.on("agent_speech_committed", lambda msg: on_speech_committed(ctx, participant))
This setup will ensure that once the bot finishes speaking, the call is terminated. Would you like more details on any part of this process? Sources: - LiveKit Slack Discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
a
@millions-hair-24956 I have a specific phrase I want it to say, then I will hang-up. How do I block on that?
g
To have the bot say a specific phrase and then hang up, you can use the 
say
method of the 
VoicePipelineAgent
to play the phrase, and then use the 
agent_speech_committed
event to trigger the hang-up action once the phrase has been spoken. Here's how you can implement this: 1. Use 
say
Method: Call the 
say
method to have the bot speak the specific phrase. 2. Listen for 
agent_speech_committed
: Register a callback for the 
agent_speech_committed
event to hang up the call after the phrase is spoken. Here's a code example:
Copy code
import asyncio
from livekit import api
from livekit.agents.pipeline import VoicePipelineAgent

async def hang_up(ctx, participant):
    lk_api = api.LiveKitAPI()
    await lk_api.room.remove_participant(
        api.RoomParticipantIdentity(
            room=ctx.room.name, identity=participant.identity
        )
    )

def on_speech_committed(ctx, participant):
    asyncio.create_task(hang_up(ctx, participant))

async def main(ctx, participant):
    agent = VoicePipelineAgent(
        vad=vad_instance,
        stt=stt_instance,
        llm=llm_instance,
        tts=tts_instance
    )

    # Say the specific phrase
    await agent.say("This is the specific phrase I want to say.", allow_interruptions=False)

    # Register the callback to hang up after speaking
    agent.on("agent_speech_committed", lambda msg: on_speech_committed(ctx, participant))

# Assuming `ctx` and `participant` are defined in your context
# asyncio.run(main(ctx, participant))
This code will ensure that the bot says the specified phrase and then hangs up the call. Would you like more details on any part of this process? Sources: - LiveKit Slack Discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
a
@millions-hair-24956 I want to detect if there hasn't been any activity and prompt the user to talk again
g
To detect inactivity and prompt the user to speak again, you can use a combination of Voice Activity Detection (VAD) and a timer. Here's a general approach: 1. Set Up VAD: Use the 
user_stopped_speaking
event to detect when the user stops speaking. 2. Start a Timer: When the user stops speaking, start a timer for a specified duration (e.g., 5 seconds). 3. Prompt the User: If the timer completes without detecting any new speech activity, use the 
say
method to prompt the user to speak again. Here's a code example:
Copy code
import asyncio
from livekit.agents.pipeline import VoicePipelineAgent

timeout_task = None
TIMEOUT_SECONDS = 5

async def prompt_user(agent):
    await agent.say("Please say something to continue.", allow_interruptions=False)

def reset_timeout(agent):
    global timeout_task
    if timeout_task and not timeout_task.done():
        timeout_task.cancel()
    timeout_task = asyncio.create_task(start_timeout(agent))

async def start_timeout(agent):
    try:
        await asyncio.sleep(TIMEOUT_SECONDS)
        await prompt_user(agent)
    except asyncio.CancelledError:
        pass

agent = VoicePipelineAgent(
    vad=vad_instance,
    stt=stt_instance,
    llm=llm_instance,
    tts=tts_instance
)

agent.on("user_stopped_speaking", lambda *args: reset_timeout(agent))
agent.on("user_started_speaking", lambda *args: reset_timeout(agent))
This setup will prompt the user to speak again if there is no activity for the specified timeout period. Would you like more details on any part of this process? Sources: - LiveKit Slack Discussion 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
a
@millions-hair-24956 but sometimes the bot might say something lengthy, I don't want to count that as part of the 5 seconds, I only want to count 5 seconds of silence
g
To ensure that the 5-second timer only counts silence and not the bot's speech, you can adjust the logic to start the timer only after both the user and the bot have stopped speaking. Here's how you can implement this: 1. Track Speaking Events: Use both 
user_stopped_speaking
and 
agent_stopped_speaking
events to determine when both the user and the bot are silent. 2. Start Timer on Silence: Only start the timer when both events have been triggered, indicating silence from both parties. 3. Prompt on Timer Completion: If the timer completes without any new speech activity, prompt the user. Here's a code example:
Copy code
import asyncio
from livekit.agents.pipeline import VoicePipelineAgent

timeout_task = None
TIMEOUT_SECONDS = 5
user_speaking = False
agent_speaking = False

async def prompt_user(agent):
    await agent.say("Please say something to continue.", allow_interruptions=False)

def reset_timeout(agent):
    global timeout_task
    if timeout_task and not timeout_task.done():
        timeout_task.cancel()
    if not user_speaking and not agent_speaking:
        timeout_task = asyncio.create_task(start_timeout(agent))

async def start_timeout(agent):
    try:
        await asyncio.sleep(TIMEOUT_SECONDS)
        await prompt_user(agent)
    except asyncio.CancelledError:
        pass

def on_user_stopped_speaking(*args):
    global user_speaking
    user_speaking = False
    reset_timeout(agent)

def on_user_started_speaking(*args):
    global user_speaking
    user_speaking = True
    reset_timeout(agent)

def on_agent_stopped_speaking(*args):
    global agent_speaking
    agent_speaking = False
    reset_timeout(agent)

def on_agent_started_speaking(*args):
    global agent_speaking
    agent_speaking = True
    reset_timeout(agent)

agent = VoicePipelineAgent(
    vad=vad_instance,
    stt=stt_instance,
    llm=llm_instance,
    tts=tts_instance
)

agent.on("user_stopped_speaking", on_user_stopped_speaking)
agent.on("user_started_speaking", on_user_started_speaking)
agent.on("agent_stopped_speaking", on_agent_stopped_speaking)
agent.on("agent_started_speaking", on_agent_started_speaking)
This setup ensures that the timer only starts when both the user and the bot are silent, and it will prompt the user if no activity is detected for 5 seconds. Would you like more details on any part of this process? Sources: - https://docs.livekit.io/agents/voice-agent/voice-pipeline/ - https://docs.livekit.io/agents/voice-agent/ 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
a
@millions-hair-24956 If
Copy code
@llm.ai_callable()
    async def detected_answering_machine(
        self,
    ):
        """
        Use this when the call reaches voicemail, AFTER you hear the voicemail greeting intro.
        """
        if isinstance(self.agent, VoicePipelineAgent):
            if self.prompt_data.is_last_call:
                await self.agent.say(self.prompt_data.last_call_message)
                self.agent.on("agent_speech_committed", lambda msg: asyncio.create_task(self.end_call()))
                return
            await self.end_call()
        else:
            return "I just heard the user's voicemail. I should handle this according to my instructions."

    @llm.ai_callable()
if this tool gets interrupted, do not hang up since we are not in voicemail, how do I do that?
g
To ensure that the tool does not hang up if it gets interrupted (indicating that it might not be a voicemail), you can add a check for interruptions before proceeding with the hang-up logic. You can use the 
interrupted
attribute of the speech handle to determine if the speech was interrupted. Here's how you can modify your code: 1. Check for Interruptions: Before calling 
end_call
, check if the speech was interrupted. 2. Proceed Based on Interruption: Only proceed with the hang-up if the speech was not interrupted. Here's a modified version of your code:
Copy code
@llm.ai_callable()
async def detected_answering_machine(self):
    """
    Use this when the call reaches voicemail, AFTER you hear the voicemail greeting intro.
    """
    if isinstance(self.agent, VoicePipelineAgent):
        if self.prompt_data.is_last_call:
            await self.agent.say(self.prompt_data.last_call_message)
            
            def on_speech_committed(msg):
                if not msg.interrupted:  # Check if the speech was not interrupted
                    asyncio.create_task(self.end_call())
            
            self.agent.on("agent_speech_committed", on_speech_committed)
            return
        await self.end_call()
    else:
        return "I just heard the user's voicemail. I should handle this according to my instructions."
This setup ensures that the call is only ended if the speech was not interrupted, indicating that it was indeed a voicemail. Would you like more details on any part of this process? Sources: - https://docs.livekit.io/agents-js/interfaces/agents.pipeline.VPAOptions.html - https://livekit-users.slack.com/archives/C07FY8WHGPM/p1733820422.673179 Note: I wasn’t able to find highly relevant data sources, but above are a few potentially relevant links. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
a
@millions-hair-24956 'ChatMessage' object has no attribute 'interrupted'",
g
The error message "'ChatMessage' object has no attribute 'interrupted'" suggests that you are trying to access an 
interrupted
attribute on a 
ChatMessage
object, which does not exist. Instead, you should be checking the 
interrupted
status on the 
SpeechHandle
object associated with the speech event. To resolve this, ensure that you are accessing the 
interrupted
attribute from the correct object. Here's how you can modify your code to check for interruptions correctly: 1. Access the Correct Object: Ensure you are using the 
SpeechHandle
object to check for interruptions. Here's an example of how you might adjust your code:
Copy code
@llm.ai_callable()
async def detected_answering_machine(self):
    """
    Use this when the call reaches voicemail, AFTER you hear the voicemail greeting intro.
    """
    if isinstance(self.agent, VoicePipelineAgent):
        if self.prompt_data.is_last_call:
            speech_handle = await self.agent.say(self.prompt_data.last_call_message)
            
            def on_speech_committed(handle):
                if not handle.interrupted:  # Check if the speech was not interrupted
                    asyncio.create_task(self.end_call())
            
            self.agent.on("agent_speech_committed", on_speech_committed)
            return
        await self.end_call()
    else:
        return "I just heard the user's voicemail. I should handle this according to my instructions."
In this example, 
speech_handle
is used to track the speech event, and its 
interrupted
attribute is checked to determine if the speech was interrupted. Would you like more details on any part of this process? Sources: - SpeechHandle Documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
Open in Slack
PreviousNext
Powered by