how can i use whisper stt that is on azure open ai service , what should i pass and what should be the structure

Thanks for reaching out, Gaurav. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i am using python and livikit agent framework . I have azure open ai where i have whisper model deployment

I'm happy to help you get started with using Whisper STT alongside Azure OpenAI Service and the LiveKit agent framework! Here are some important details to guide you through: 1. Using Azure OpenAI STT: • LiveKit supports Azure OpenAI STT with the

openai.STT.with_azure()

method. However, it’s good to know that the current sources don’t confirm direct support for the Whisper model (

whisper-1

) specifically under Azure OpenAI. • Typically, Azure OpenAI uses models like

gpt-4o-transcribe

or

gpt-4o-mini-transcribe

for speech-to-text tasks, which are great alternatives. 2. Integration Steps: 1. First, install the required plugin by running:

pip install "livekit-agents[openai]~=1.2"

2. Next, set your environment variables with your Azure OpenAI credentials:

AZURE_OPENAI_API_KEY=<your-api-key>
AZURE_OPENAI_ENDPOINT=<your-azure-endpoint>
AZURE_OPENAI_AD_TOKEN=<optional-ad-token>

1. Then, initialize the STT in Python like this:

from livekit.plugins import openai
from livekit.agents import AgentSession

stt = openai.STT.with_azure(
    model="gpt-4o-transcribe",  # or "gpt-4o-mini-transcribe"
    language="en",
    azure_deployment="your-transcription-deployment-name",
    api_version="2024-09-01-preview"  # Use latest supported version
)

session = AgentSession(
    stt=stt,
    # ... other components (LLM, TTS)
)

3. Alternative Options: • If you're specifically looking for Whisper-like transcription, you might want to try the standard OpenAI STT API using

openai.STT(model="whisper-1")

. • Another option is to use Azure Speech STT (which is separate from Azure OpenAI) through the

livekit-plugins-azure

plugin, offering real-time and highly accurate transcription. Please feel free to reach out if you have any questions or need more help with these steps—I’m here to support you! Sources: OpenAI STT integration guide | LiveKit Docs | OpenAI integration guide | LiveKit Docs | OpenAI integration guide | LiveKit Docs | examples/other/transcription/README.md | Azure integration guide | LiveKit Docs