i move azure api to GA

base_url = (
    azure_openai_settings.endpoint.replace("https://", "wss://").rstrip("/")
    + "/openai/v1/realtime"
)

llm = RealtimeModel(
    model=agent_settings.LLM_MODEL,
    voice=agent_settings.LLM_VOICE,
    base_url=f"{base_url}?api-key={azure_openai_settings.api_key}",
    api_key="value does not matter",
    modalities=["text"],
    turn_detection=realtime.realtime_audio_input_turn_detection.SemanticVad(
        type="semantic_vad",
        eagerness="auto",
        create_response=True,
        interrupt_response=True,
    ),
)

and now get OpenAI Realtime API failed to transcribe input audio

Thanks for reaching out, Aleh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

114035.185 ERROR livekit.…ns.openai OpenAI Realtime API failed to transcribe input audio {"error": "Error(code=None, message=\"Input transcription failed for item 'item_DSIkasrkovJIE1BBTu9Q5'.\", param=None, type='server_error')"}

Thanks — I’m responding about the transcription error you pasted ("Input transcription failed for item 'item_DSIkasrkovJIE1BBTu9Q5'"). Please try these focused troubleshooting steps and reply with the requested logs/config if it still fails: 1. Switch VAD mode ◦ Azure Realtime supports Server VAD only. Replace

SemanticVad

with server-side VAD or remove

SemanticVad

and let the service use its default Server VAD. 1. Use the Azure helper (do not hand-build the websocket URL) ◦ Use the SDK’s

with_azure()

or the Azure-specific factory so endpoint, API version, and auth are set correctly (avoid embedding the key in the query string). 1. Verify environment variables ◦ Ensure

AZURE_OPENAI_ENDPOINT

,

AZURE_OPENAI_API_KEY

, and

OPENAI_API_VERSION

are set and match your GA deployment. 1. Check audio format & framing ◦ Confirm you’re sending supported PCM/Opus audio, correct sample rate, and proper chunk framing per the realtime SDK you’re using. 1. Capture and share these logs ◦ The websocket handshake and server responses around the failure. ◦ Any OpenAI/Azure HTTP status codes or error fields (beyond the single server_error message). ◦ A short trace of the audio items around the failed item (the 10–20 log lines before/after the error). If you can’t change the config immediately, share: • The exact

RealtimeModel

initialization code you’re using now (with secrets redacted), and • 30–50 lines of logs showing the websocket connect, auth, and the moment the transcription fails. Send those and I’ll give exact code replacements and next steps. , with interactive elements

before i have

@staticmethod
def _setup_voice_components(
    agent_settings: AgentSettings,
) -> tuple[TTS, RealtimeModel | RealtimeModelBeta]:
    """Setup TTS and LLM components."""
    vibe_instructions = vibe_instruction.invoke({}).text

    tts = TTS.with_azure(
        azure_endpoint=azure_openai_tts_settings.endpoint,
        api_key=azure_openai_tts_settings.api_key,
        model=agent_settings.TTS_MODEL,
        azure_deployment=agent_settings.TTS_MODEL,
        voice=agent_settings.TTS_VOICE,
        speed=agent_settings.TTS_SPEED,
        instructions=vibe_instructions,
    )

    llm = RealtimeModel.with_azure(
        azure_deployment=agent_settings.LLM_MODEL,
        voice=agent_settings.LLM_VOICE,
        turn_detection=TurnDetection(
            type="semantic_vad",
            eagerness="auto",
            create_response=True,
            interrupt_response=True,
        ),
    )

    return tts, llm

now it is

@staticmethod
def _setup_voice_components(
    agent_settings: AgentSettings,
) -> tuple[TTS, RealtimeModel]:
    """Setup TTS and LLM components."""
    vibe_instructions = vibe_instruction.invoke({}).text

    tts = TTS.with_azure(
        azure_endpoint=azure_openai_tts_settings.endpoint,
        api_key=azure_openai_tts_settings.api_key,
        api_version=azure_openai_tts_settings.api_version,
        model=agent_settings.TTS_MODEL,
        azure_deployment=agent_settings.TTS_MODEL,
        voice=agent_settings.TTS_VOICE,
        speed=agent_settings.TTS_SPEED,
        instructions=vibe_instructions,
    )

    base_url = (
        azure_openai_settings.endpoint.replace("https://", "wss://").rstrip("/")
        + "/openai/v1/realtime"
    )

    llm = RealtimeModel(
        model=agent_settings.LLM_MODEL,
        voice=agent_settings.LLM_VOICE,
        base_url=f"{base_url}?api-key={azure_openai_settings.api_key}",
        api_key="value does not matter",
        input_audio_noise_reduction="near_field",
        turn_detection=realtime.realtime_audio_input_turn_detection.SemanticVad(
            type="semantic_vad",
            create_response=True,
            eagerness="auto",
            interrupt_response=True,
        ),
    )

    return tts, llm

i rollback to previous and was able to get user transcript, how i can adopt new one

Thanks — I’m responding to your code change and the "Input transcription failed" error after moving to the GA setup. What caused it • Azure OpenAI Realtime only supports Server VAD. Using

SemanticVad

causes transcription to fail. • When using the GA/websocket setup you must also explicitly enable model transcription via

input_audio_transcription

. What to change (minimal, actionable) 1. Replace

SemanticVad

with

ServerVad

. 2. Add

input_audio_transcription

to the

RealtimeModel

so the model returns transcripts. 3. Prefer the SDK Azure helper where possible; if you manually build the wss URL, URL-encode the key. Example replacement (insert into your new GA code):

from livekit.plugins.openai import realtime

turn_detection = realtime.ServerVad(
    type="server_vad",
    threshold=0.5,
    prefix_padding_ms=300,
    silence_duration_ms=200,
    create_response=True,
    interrupt_response=True,
)

llm = RealtimeModel(
    model=agent_settings.LLM_MODEL,
    voice=agent_settings.LLM_VOICE,
    base_url=f"{azure_openai_settings.endpoint.replace('https://','wss://').rstrip('/')}/openai/v1/realtime?api-key={azure_openai_settings.api_key}",
    api_key="ignored",
    turn_detection=turn_detection,
    input_audio_transcription=realtime.AudioTranscription(model="whisper-1"),
    input_audio_noise_reduction="near_field",
)

Quick checks if it still fails • Confirm `AZURE_OPENAI_ENDPOINT`/`AZURE_OPENAI_API_KEY`/`OPENAI_API_VERSION` match your GA deployment. • Ensure your audio chunks are in a supported format (PCM/Opus) and framing is correct. • URL-encode the api key if it contains special characters. If you want, paste 20–30 lines of the websocket handshake logs (with secrets redacted) and I’ll scan for auth/handshake or transcription error codes. Sources: Azure OpenAI Realtime API and LiveKit | LiveKit Documentation | Realtime models overview | LiveKit Documentation | examples/voice_agents | livekit/livekit_composite | DeepWiki, with interactive elements

i tried

turn_detection=TurnDetection(
    type="semantic_vad",
    eagerness="auto",
    create_response=True,
    interrupt_response=True,
),

and still get transcript errors

Thanks — I’m responding to your note that

SemanticVad

still produces transcript errors. Short answer: Azure does not support

SemanticVad

. Using

SemanticVad

will fail; you must use Server VAD and explicitly enable input transcription. Please replace your turn detection with Server VAD and add

input_audio_transcription

. Example minimal change:

from livekit.plugins.openai import realtime

turn_detection = realtime.ServerVad(
    type="server_vad",
    silence_duration_ms=200,
    prefix_padding_ms=300,
    create_response=True,
    interrupt_response=True,
)

llm = RealtimeModel(
    model=agent_settings.LLM_MODEL,
    voice=agent_settings.LLM_VOICE,
    base_url=f"{azure_openai_settings.endpoint.replace('https://','wss://').rstrip('/')}/openai/v1/realtime?api-key={azure_openai_settings.api_key}",
    api_key="ignored",
    turn_detection=turn_detection,
    input_audio_transcription=realtime.AudioTranscription(model="whisper-1"),
    input_audio_noise_reduction="near_field",
)

If that still fails, please paste: • 20–30 lines of the websocket handshake and the lines immediately surrounding the transcription error (redact keys), • the audio format you’re sending (PCM/Opus, sample rate, chunk size). I’ll inspect those and give next steps. , with interactive elements

AttributeError: module 'openai.types.realtime' has no attribute 'ServerVad'

Thanks — I’m responding to the AttributeError (

openai.types.realtime

has no attribute

ServerVad

) and how to fix your migration. What happened • The GA SDK no longer exposes

ServerVad

at that path. Use the generic

TurnDetection

object with

type="server_vad"

(the SDK’s TurnDetection lives under the beta realtime session types). What to change (minimal): 1. Import and use

TurnDetection

with

type="server_vad"

instead of

ServerVad

or

semantic_vad

. 2. Ensure you explicitly enable transcription with

input_audio_transcription

. Example minimal replacement:

from openai.types.beta.realtime.session import TurnDetection
from livekit.plugins.openai import realtime

turn_detection = TurnDetection(
    type="server_vad",
    prefix_padding_ms=300,
    silence_duration_ms=200,
    # optional: threshold=0.5,
    create_response=True,
    interrupt_response=True,
)

llm = RealtimeModel(
    model=agent_settings.LLM_MODEL,
    voice=agent_settings.LLM_VOICE,
    base_url=f"{azure_openai_settings.endpoint.replace('https://','wss://').rstrip('/')}/openai/v1/realtime?api-key={azure_openai_settings.api_key}",
    api_key="ignored",
    turn_detection=turn_detection,
    input_audio_transcription=realtime.AudioTranscription(model="whisper-1"),
    input_audio_noise_reduction="near_field",
)

Quick checks if you still see transcription errors • Confirm you have the latest `livekit-agents[openai]`/OpenAI realtime packages installed. • Verify

AZURE_OPENAI_ENDPOINT

,

AZURE_OPENAI_API_KEY

, and

OPENAI_API_VERSION

. • If errors persist, paste 20–30 lines of the websocket handshake + the transcription error log (redact keys) and the audio format (PCM/Opus, sample rate, chunk size). Sources: Azure OpenAI Realtime API and LiveKit | LiveKit Documentation | OpenAI Realtime API plugin guide | LiveKit Documentation, with interactive elements

120535.248 ERROR livekit.…ns.openai OpenAI Realtime API failed to transcribe input audio {"error": "Error(code=None, message=\"Input transcription failed for item 'item_DSJ8nuN43uxyTUX2zZGps'.\",

turn_detection=TurnDetection(
    type="server_vad",
    prefix_padding_ms=300,
    silence_duration_ms=200,
    # optional: threshold=0.5,
    create_response=True,
    interrupt_response=True,
)

Thanks — I’m responding to the repeated "Input transcription failed" error you just posted. Next steps to try (short checklist): 1. Ensure transcription is explicitly enabled ◦ Add

input_audio_transcription=realtime.AudioTranscription(model="whisper-1")

to the

RealtimeModel

(if you haven't already). 1. Confirm Server VAD usage (you already set

TurnDetection(type="server_vad")

) — good. Keep

silence_duration_ms

low (100–300ms) while debugging. 2. Verify audio format & framing ◦ Are you sending Opus or PCM? Confirm sample rate (e.g., 16k/48k), correct chunk sizes, and that each audio item is properly encoded and base64/frames per your SDK. ◦ If using client-side capture, try a short known-good audio file (mono PCM 16-bit 16k) to rule out capture encoding issues. 1. Validate GA endpoint/auth ◦ Use the wss URL form you have, but URL-encode the api key. Confirm

OPENAI_API_VERSION

matches your GA deployment (if required). 1. Capture the relevant logs (if error persists) ◦ Paste 20–30 lines around the websocket handshake and the transcription failure (redact keys). ◦ Include the audio item id (e.g., item_DSJ8nuN43uxyTUX2zZGps) and the preceding logs for that item (sending, content-length, encoding). If you want, paste: • The full

RealtimeModel

constructor you’re using (redact keys), • One sample websocket log block (20–30 lines), • The audio format (codec, sample rate, mono/stereo, chunk size). I’ll inspect those and give exact fixes. , with interactive elements

llm = RealtimeModel(
    model=agent_settings.LLM_MODEL,
    voice=agent_settings.LLM_VOICE,
    base_url=f"{base_url}?api-key={azure_openai_settings.api_key}",
    input_audio_transcription=realtime_beta.session.InputAudioTranscription(
        model="gpt-4o-transcribe"
    ),
    api_key="value does not matter",
    input_audio_noise_reduction="near_field",
    turn_detection=realtime.realtime_audio_input_turn_detection.SemanticVad(
        type="semantic_vad",
        create_response=True,
        eagerness="auto",
        interrupt_response=True,
    ),
)

120908.898 ERROR livekit.…ns.openai OpenAI Realtime API failed to transcribe input audio {"error": "Error(code=None, message=\"Input transcription failed for item 'item_DSJCEbLz0vsMkjnMWgYX0'.\", param=None, type='server_error')"}

Thanks — I’m responding to your latest config and the continuing "Input transcription failed" errors. Two immediate problems in the snippet you posted: • You’re still using

SemanticVad

— Azure does not support semantic VAD; use

TurnDetection(type="server_vad")

. • You mixed beta/GA types: use the GA/compatible imports for

TurnDetection

and

InputAudioTranscription

rather than the semantic VAD class from the realtime path. Do this minimal, exact change: 1) Import and use

TurnDetection

with

type="server_vad"

and enable transcription with the proper InputAudioTranscription type:

from openai.types.beta.realtime.session import TurnDetection
from livekit.plugins.openai import realtime
from openai.types.beta.realtime.session import InputAudioTranscription

llm = RealtimeModel(
    model=agent_settings.LLM_MODEL,
    voice=agent_settings.LLM_VOICE,
    base_url=f"{base_url}?api-key={azure_openai_settings.api_key}",
    api_key="ignored",
    input_audio_transcription=InputAudioTranscription(model="gpt-4o-transcribe"),
    input_audio_noise_reduction="near_field",
    turn_detection=TurnDetection(
        type="server_vad",
        prefix_padding_ms=300,
        silence_duration_ms=200,
        create_response=True,
        interrupt_response=True,
    ),
)

2) Quick checks if errors continue: • Ensure the API key in

base_url

is URL-encoded. • Confirm your audio format: mono PCM/Opus, expected sample rate (e.g., 16k/48k), correct chunk framing. • Verify

AZURE_OPENAI_ENDPOINT

,

AZURE_OPENAI_API_KEY

, and

OPENAI_API_VERSION

match your GA deployment. If it still fails, paste: • The full RealtimeModel constructor you now have (redact keys), • 20–30 lines of websocket handshake + the lines around the transcription error, • One sample audio item’s format (codec, sample rate, mono/stereo, chunk size). I’ll inspect those and give the next concrete fix. , with interactive elements