LiveKit Community

am i able to add a custom STT to my multimodal openai agent? for example, can i use deepgram with my multimodal agent?

Thanks for reaching out, Rhythm. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.


You can use Deepgram as a custom speech-to-text (STT) provider with your multimodal OpenAI agent in LiveKit. The LiveKit Agents framework supports combining different providers, so you can use Deepgram for STT and OpenAI for LLM functionality within the same agent session.
To integrate Deepgram STT with your agent:
1. Install the Deepgram plugin: `pip install "livekit-agents[deepgram]~=1.0"`
2. Set your Deepgram API key as the `DEEPGRAM_API_KEY` environment variable
3. Configure your agent session to use Deepgram for STT, for example:
```from livekit.plugins import deepgram

session = AgentSession(
    stt = deepgram.STT(
        model="nova-3",
    ),
    # ... llm, tts, etc.
)
```
You can also adjust configuration parameters such as the model, language, and endpointing settings. This setup allows you to mix and match the best STT, LLM, and TTS providers for your use case.
Let me know if you need details on other STT providers or further configuration options.
