https://livekit.io logo
Join Slack
Powered by
Hello, please help me, I want to change AI agent ...
# ask-ai
e
Hello, please help me, I want to change AI agent working so: it starts speaking and then after words for example "quiz" it sends message to frontend to start quiz. Could anyone give me ideas?
t
Thanks for reaching out, Tetiana. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Could you please share a bit more about whether you're using LiveKit's AI agent alongside a particular speech recognition or natural language processing (NLP) system? Also, I’d love to understand if you’re looking for some help with integrating this feature into your current frontend application. I’m here to support you every step of the way!
To help you trigger a frontend action like starting a quiz when your AI agent hears specific words, you can adjust your agent’s behavior to spot keywords in real-time speech transcription. Let me guide you through how to do this: 1. Override the 
stt_node
Method: By overriding the 
stt_node
method in your agent class, you can handle speech-to-text (STT) events directly. This way, you can check the incoming transcriptions and look out for particular keywords.
Copy code
async def stt_node(self, text: AsyncIterable[str], model_settings: Optional[dict] = None):
    parent_stream = super().stt_node(text, model_settings)
    keywords = ["quiz"]

    async def process_stream():
        async for event in parent_stream:
            if hasattr(event, 'type') and str(event.type) == "SpeechEventType.FINAL_TRANSCRIPT" and event.alternatives:
                transcript = event.alternatives[0].text
                for keyword in keywords:
                    if keyword.lower() in transcript.lower():
                        # Trigger your frontend action here
                        print(f"Keyword detected: '{keyword}'")
            yield event
    return process_stream()
2. Trigger Frontend Actions: Once your agent picks up the keyword (like "quiz"), you can have it trigger a custom action—maybe sending a message to your frontend app. This could be done via WebSocket, HTTP requests, or whichever communication method works best for you. 3. Optional: Use Keyword Boosting (Google STT): If you’re using Google STT, there’s a handy feature to boost recognition accuracy for specific keywords by giving them boost values. Just remember, you’ll still want to process the output in 
stt_node
to detect and respond to those keywords.
Copy code
stt=google.STT(keywords=[("quiz", 2.0)])
If you’d like, I’m here to help you with any part of implementing this—just let me know! Sources: Pipeline nodes and hooks | LiveKit Docs | Pipeline nodes and hooks | LiveKit Docs | Docs::Agents | livekit-agents/livekit/agents/voice/agent_activity.py | Function calling with Voice Agents | LiveKit Docs
e
I've got error async def stt_node(self, text: AsyncIterable[str], model_settings: Optional[dict] = None): ^^^^^^^^^^^^^ NameError: name 'AsyncIterable' is not defined
t
It looks like the error is happening because 
AsyncIterable
isn’t defined in the global namespace by default. No worries, though! To fix this, you just need to import it from the 
typing
module. You can do this by adding the following line at the very top of your Python file:
Copy code
from typing import AsyncIterable
This little step will make sure 
AsyncIterable
is recognized and ready to use in your type hints. If you run into any other questions or need a hand, feel free to reach out—I’m here to help! Sources: Text-to-speech (TTS) integrations | LiveKit Docs | Pipeline nodes and hooks | LiveKit Docs | Docs::Agents | Pipeline nodes and hooks | LiveKit Docs
It looks like the error is happening because 
AsyncIterable
isn’t defined in the global namespace by default. No worries though! To fix this, you just need to import it from the 
typing
module. You can do this by adding the following line at the very top of your Python file:
Copy code
from typing import AsyncIterable
This little step will make sure 
AsyncIterable
is recognized and ready to use in your type hints. If you run into any other bumps along the way, feel free to reach out—I’m here to help!
e
how fix async def stt_node(self, text: AsyncIterable[str], model_settings: Optional[dict] = None): ^^^^^^^^ NameError: name 'Optional' is not defined
t
It looks like the error is happening because 
Optional
isn’t defined in the global namespace by default. No worries though! To fix this, you just need to import it from the 
typing
module. You can do this by adding the following line at the very top of your Python file:
Copy code
from typing import Optional
This little step will make sure 
Optional
is recognized and ready to use in your type hints. If you run into any other questions or need a hand with something else, feel free to reach out—I’m here to help!
e
the code given me doesn't work properly. I didn't get "quiz" in console
t
It looks like the keyword "quiz" isn’t showing up in your console, so let’s walk through some friendly troubleshooting steps together to get this sorted out: 1. Enable Logging: First, let’s make sure your logging is set up to catch those detection events. You can add this to your code:
Copy code
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
Then, log the transcript to see exactly what’s being processed:
Copy code
<http://logger.info|logger.info>(f"Transcript received: '{transcript}'")
2. Verify Keyword Matching Logic: Next, double-check that your keyword detection logic is spot on. For example:
Copy code
if "quiz" in transcript.lower():
    <http://logger.info|logger.info>("Keyword 'quiz' detected!")
This will help confirm if the spoken word matches what you’re looking for. 3. Check STT Output: It’s important to make sure your Speech-to-Text (STT) engine is providing final transcripts. You can add some logging like this:
Copy code
if hasattr(event, 'type') and str(event.type) == "SpeechEventType.FINAL_TRANSCRIPT":
    <http://logger.info|logger.info>("Final transcript received.")
4. Adjust STT Configuration: If you’re working with Deepgram, double-check your configuration. For instance, Nova-3 models use 
keyterms
instead of `keywords`:
Copy code
stt=deepgram.STT(keyterms=["quiz"])
If any of these steps feel tricky or you want me to help you dig deeper, just let me know—I’m here to support you! Sources: README.md | livekit-agents/livekit/agents/voice/agent_activity.py | Pipeline nodes and hooks | LiveKit Docs
It looks like the keyword "quiz" isn't being detected in the console, so let's walk through some steps together to help you debug and fix this: 1. Enable Logging: First, let's make sure logging is set up correctly to capture all detection events. You can add this to your code:
Copy code
import logging
logging.basicConfig(level=logging.DEBUG)
This will give you a clear view of all events and transcripts being processed, which is super helpful. 2. Log Transcripts: Next, let's log the actual transcripts you receive to check if the keyword "quiz" is appearing:
Copy code
<http://logger.info|logger.info>(f"Transcript received: '{transcript}'")
Try placing this inside your 
stt_node
method where the transcripts are handled. 3. Check Keyword Matching: Make sure your keyword detection logic is spot on. For example:
Copy code
if "quiz" in transcript.lower():
    <http://logger.info|logger.info>("Keyword 'quiz' detected!")
Double-check that the spoken audio contains the exact word "quiz" or something very close to it. 4. Verify STT Output: It's important to confirm that your STT engine is returning final transcripts. Usually, keywords show up in 
FINAL_TRANSCRIPT
events. You can log the event type like this:
Copy code
<http://logger.info|logger.info>(f"Event type: {event.type}")
5. Adjust STT Configuration: If you're using a specific STT provider like Deepgram, make sure it's set up to recognize your keyword. For instance, with Deepgram Nova-3 models, you can specify keyterms like this:
Copy code
stt=deepgram.STT(keyterms=["quiz"])
Give these steps a try, and please feel free to reach out if you need any more help. I'm here to support you!
e
stt=deepgram.STT(keyterms=["quiz"]) ^^^^^^^^ NameError: name 'deepgram' is not defined
t
It looks like the error is happening because the 
deepgram
module hasn't been imported in your code yet. No worries—let's walk through the steps together to get this sorted out: 1. Install the Deepgram Plugin: First, make sure the Deepgram plugin is installed by running this command:
Copy code
pip install "livekit-agents[deepgram]~=1.2"
2. Import the Deepgram Module: Next, add this import statement at the very top of your Python file:
Copy code
from livekit.plugins import deepgram
3. Set the API Key: Don't forget to set your Deepgram API key as an environment variable like this:
Copy code
DEEPGRAM_API_KEY=your_api_key_here
Once you've done these steps, you should be all set to use 
deepgram.STT
without any issues. If you run into any trouble or have more questions, please feel free to reach out—I'm here to help! Sources: agent.py | pipeline-tts/elevenlabs_change_language.py | Deepgram TTS integration guide | LiveKit Docs | Deepgram STT integration guide | LiveKit Docs | livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/__init__....
It looks like the error is happening because the 
deepgram
module hasn’t been imported in your code yet. No worries—let’s walk through the steps to get this sorted out together: 1. Install the Deepgram Plugin: First, make sure the LiveKit Deepgram plugin is installed by running this command:
Copy code
pip install "livekit-agents[deepgram]~=1.2"
2. Import the Deepgram Module: Next, add this import statement right at the top of your Python file:
Copy code
from livekit.plugins import deepgram
3. Set the API Key: Don’t forget to set your Deepgram API key as an environment variable like this:
Copy code
DEEPGRAM_API_KEY=your_api_key_here
Once you’ve done these steps, you should be all set to use 
deepgram.STT
without any hiccups. If you run into any other questions or need a hand, just let me know—I’m here to help!
Hi there! To help your agent detect the keyword "quiz" and print it to the console, let’s walk through some steps together to debug and improve your setup: 1. Verify Logging Configuration: First, make sure your logging is configured to show messages in the console. You can do this by adding the following at the top of your file:
Copy code
import logging
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
Also, instead of using 
print()
, try using 
<http://logger.info|logger.info>()
for better logging—it really helps keep things organized. 2. Debug the 
stt_node
Method: Let’s ensure your 
stt_node
method is correctly handling the transcripts. Here’s an updated version you can try:
Copy code
async def stt_node(self, text: AsyncIterable[str], model_settings: Optional[dict] = None):
    parent_stream = super().stt_node(text, model_settings)
    if parent_stream is None:
        return None

    keywords = ["quiz"]

    async def process_stream():
        async for event in parent_stream:
            if hasattr(event, 'type') and str(event.type) == "SpeechEventType.FINAL_TRANSCRIPT" and event.alternatives:
                transcript = event.alternatives[0].text
                <http://logger.info|logger.info>(f"Transcript received: '{transcript}'")  # Log the transcript
                for keyword in keywords:
                    if keyword.lower() in transcript.lower():
                        <http://logger.info|logger.info>(f"Keyword detected: '{keyword}'")  # Log keyword detection
            yield event

    return process_stream()
3. Test with Actual Speech: When you test, try saying the word "quiz" clearly and see if it shows up in the transcript. If it doesn’t, it might be worth double-checking your STT engine settings. 4. Verify STT Configuration: If you’re using Deepgram, make sure it’s properly set up. For example:
Copy code
stt = deepgram.STT(keyterms=["quiz"])
Also, confirm that your Deepgram API key is correctly set as an environment variable. If you run into any issues or want to explore this further, please don’t hesitate to reach out—I’m here to help! Sources: Docs::Agents | Deepgram integration guide | LiveKit Docs | Deepgram integration guide | LiveKit Docs | home_assistant/homeautomation.py | Pipeline nodes and hooks | LiveKit Docs
Hi there! To help your agent detect the keyword "quiz" and print it to the console, let’s walk through some steps together to debug and refine your implementation: 1. Verify Logging Configuration: First, make sure logging is set up to show messages in the console. You can add this at the top of your file:
Copy code
import logging
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
2. Debug the 
stt_node
Method: Try updating your 
stt_node
method to log the transcript so you can see if the keyword "quiz" is being detected:
Copy code
async def stt_node(self, text: AsyncIterable[str], model_settings: Optional[dict] = None):
    parent_stream = super().stt_node(text, model_settings)
    keywords = ["quiz"]

    async def process_stream():
        async for event in parent_stream:
            if hasattr(event, 'type') and str(event.type) == "SpeechEventType.FINAL_TRANSCRIPT" and event.alternatives:
                transcript = event.alternatives[0].text
                print(f"Transcript received: {transcript}")  # Log the transcript
                for keyword in keywords:
                    if keyword.lower() in transcript.lower():
                        print(f"Keyword detected: '{keyword}'")  # Print detected keyword
            yield event
    return process_stream()
3. Test with Speech Containing "Quiz": Make sure the audio you’re testing with actually includes the word "quiz" and check if it shows up in the logged transcript. 4. Check STT Configuration: If you’re using Deepgram, double-check that it’s set up correctly. For example:
Copy code
stt = deepgram.STT(keyterms=["quiz"])
Just a heads-up: the 
keyterms
parameter helps improve recognition but doesn’t directly trigger actions. The keyword detection happens in your 
stt_node
method. If you run into any issues or want me to walk through any part in more detail, please don’t hesitate to ask—I’m here to help!
Open in Slack
PreviousNext
Powered by