I am using the Grok Voice Agent API following there docs in the live kit but still getting the error what i am missing here it was pointing to the Open ai Realtime why? os.environ["XAI_API_KEY"] = settings.XAI_API_KEY session = AgentSession( llm=xai.realtime.RealtimeModel( voice="Ara", api_key=settings.XAI_API_KEY, ), vad=ctx.proc.userdata["vad"], preemptive_generation=True, max_tool_steps=1, user_away_timeout=180, min_interruption_duration=0.35, min_interruption_words=2, min_endpointing_delay=0.3, max_endpointing_delay=0.8, false_interruption_timeout=1.0, turn_detection="vad", ) error {"message": "unhandled exception while running the job task", "level": "ERROR", "name": "livekit.agents", "exc_info": "Traceback (most recent call last):\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\opentelemetry\\util\\_decorator.py\", line 71, in async_wrapper\n return await func(*args, **kwargs) # type: ignore\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\agents\\ipc\\job_proc_lazy_main.py\", line 286, in _traceable_entrypoint\n await self._job_entrypoint_fnc(job_ctx)\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\app\\agent_worker.py\", line 394, in entrypoint\n llm=xai.realtime.RealtimeModel(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\xai\\realtime\\realtime_model.py\", line 53, in __init__\n super().__init__(\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 323, in __init__\n raise ValueError(\nValueError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable", "room": "room-24a741b4-f29c-4669-bc56-79eb8242fa5c-1c98fdd3", "timestamp": "2025-12-18T092216.942271+00:00"}

Thanks for reaching out, khushal_ZC. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

but i want to used this not open ai reaktime? https://docs.livekit.io/agents/integrations/xai/

Thanks — I understand you want to use the Grok Voice Agent (XAI) and not OpenAI Realtime. You're seeing OpenAI referenced because the XAI plugin in your runtime instantiates or reuses the OpenAI Realtime client internally, and that internal client requires an OpenAI API key to be present even when you use the Grok model. To get your Grok setup working, try the following: 1) Set both environment variables before creating the `AgentSession`:

import os
os.environ["XAI_API_KEY"] = settings.XAI_API_KEY
os.environ["OPENAI_API_KEY"] = settings.XAI_API_KEY

Then create your session as you already do. 2) Confirm the key string is correct and visible to the process that runs the code (set it in the shell, Docker env, or service config that starts the app). 3) Verify plugin/SDK versions match the docs page you linked — if you recently upgraded, the plugin may require the OpenAI key to remain present. If you want, tell me how you run the app (local, Docker, systemd, etc.) and I’ll show the exact place to set the env vars for that environment or suggest a safer way to inject the key.

check this issue after doing your chnages? my code os.environ["XAI_API_KEY"] = settings.XAI_API_KEY os.environ["OPENAI_API_KEY"] = settings.XAI_API_KEY session = AgentSession( llm=xai.realtime.RealtimeModel( voice="Ara", api_key=settings.XAI_API_KEY, ), vad=ctx.proc.userdata["vad"], preemptive_generation=True, max_tool_steps=1, user_away_timeout=180, min_interruption_duration=0.35, min_interruption_words=2, min_endpointing_delay=0.3, max_endpointing_delay=0.8, false_interruption_timeout=1.0, turn_detection="vad", ) error 88+00:00"} {"message": "OpenAI Realtime API connection failed, retrying in 2.0s", "level": "WARNING", "name": "livekit.plugins.openai", "exc_info": "Traceback (most recent call last):\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 809, in _create_ws_conn\n return await asyncio.wait_for(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Users\\Admin\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\asyncio\\tasks.py\", line 510, in wait_for\n return await fut\n ^^^^^^^^^\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\aiohttp\\client.py\", line 1065, in _ws_connect\n raise WSServerHandshakeError(\naiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url='wss://api.x.ai/v1/realtime?model=grok-4-1-fast-non-reasoning'\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 751, in _main_task\n ws_conn = await self._create_ws_conn()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 814, in _create_ws_conn\n raise APIConnectionError(\"OpenAI Realtime API client connection error\") from e\nlivekit.agents._exceptions.APIConnectionError: OpenAI Realtime API client connection error (body=None, retryable=True)", "attempt": 1, "max_retries": 3, "room": "room-24a741b4-f29c-4669-bc56-79eb8242fa5c-b7987c62", "timestamp": "2025-12-18T093255.049921+00:00"} {"message": "Starting recording for the session", "level": "INFO", "name": "agent_worker", "room": "room-24a741b4-f29c-4669-bc56-79eb8242fa5c-b7987c62", "timestamp": "2025-12-18T093255.244507+00:00"} {"message": "Attempting to start recording for : TR_AM4CgcTwbxiuQh", "level": "INFO", "name": "agent_worker", "room": "room-24a741b4-f29c-4669-bc56-79eb8242fa5c-b7987c62", "timestamp": "2025-12-18T093255.244507+00:00"} {"message": "draining worker", "level": "INFO", "name": "livekit.agents", "id": "AW_oS28KLrdPmS5", "timeout": 1800, "timestamp": "2025-12-18T093256.974751+00:00"} {"message": "OpenAI Realtime API connection failed, retrying in 2.0s", "level": "WARNING", "name": "livekit.plugins.openai", "exc_info": "Traceback (most recent call last):\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 809, in _create_ws_conn\n return await asyncio.wait_for(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Users\\Admin\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\asyncio\\tasks.py\", line 510, in wait_for\n return await fut\n ^^^^^^^^^\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\aiohttp\\client.py\", line 1065, in _ws_connect\n raise WSServerHandshakeError(\naiohttp.client_exceptions.WSServerHandshakeError: 401, message='Invalid response status', url='wss://api.x.ai/v1/realtime?model=grok-4-1-fast-non-reasoning'\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 751, in _main_task\n ws_conn = await self._create_ws_conn()\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"C:\\Users\\Admin\\Downloads\\ai-backend-main\\ai-backend\\venv\\Lib\\site-packages\\livekit\\plugins\\openai\\realtime\\realtime_model.py\", line 814, in _create_ws_conn\n raise APIConnectionError(\"OpenAI Realtime API client connection error\") from e\nlivekit.agents._exceptions.APIConnectionError: OpenAI Realtime API client connection error (body=None, retryable=True)", "attempt": 2, "max_retries": 3, "room": "room-24a741b4-f29c-4669-bc56-79eb8242fa5c-b7987c62", "timestamp": "2025-12-18T093257.307383+00:00"}

You're still hitting a 401 from the XAI websocket — the connection to wss://api.x.ai/v1/realtime is being rejected for authentication. I’m responding about why that happened after you set both env vars and how to move forward. What the logs show • The websocket handshake to XAI returned 401 (Invalid response status). • LiveKit’s plugin reuses the OpenAI realtime client code, so logs mention "OpenAI Realtime API connection failed" even though the failure is against XAI's endpoint. • A 401 means the XAI realtime endpoint rejected your credentials or your account lacks realtime access. What to try next (quick steps) 1. Remove the OPENAI_API_KEY assignment — don’t set OPENAI_API_KEY to your XAI key: ◦ Keep only:

os.environ["XAI_API_KEY"] = settings.XAI_API_KEY

◦ Or pass

api_key=settings.XAI_API_KEY

to

RealtimeModel

(that param takes precedence). 1. Verify the XAI key itself: ◦ Run:

curl -H "Authorization: Bearer $XAI_API_KEY" <https://api.x.ai/v1/models>

◦ 200 = valid REST access; 401 = key invalid/expired. 1. Confirm you have Realtime access: ◦ XAI’s websocket realtime API is not publicly available to everyone. If your key works for REST but the websocket returns 401, your account likely doesn’t have realtime privileges. 1. Use the supported path now (if you don’t have realtime access): ◦ Switch to the standard XAI LLM integration:

LLM.with_x_ai(model=..., api_key=...)

+ separate STT and TTS plugins. That approach works today without websocket realtime access. If you want, tell me: • Does the curl test return 200 or 401? • How are you running the app (local shell, Docker, or service)? I’ll guide you from there (exact command to test, or how to switch to the supported LLM-with-STT/TTS setup). Sources: xAI LLM plugin guide | LiveKit Docs

give me correct code to used for the Grok realtime now please

You asked for the correct code to use Grok realtime — I’m responding with the supported approach and why a websocket "Grok realtime" path won't work today. Short answer • There is no public Grok realtime (WebSocket) API available to use with LiveKit Agents today. • Use the supported XAI (Grok) LLM integration plus STT and TTS (this gives voice agent behavior without a private realtime websocket). Working Python example (supported)

import os
from livekit.agents import AgentSession
from livekit.plugins import openai as openai_plugin
from livekit.plugins import assemblyai, cartesia, silero

# Ensure your XAI key is set
os.environ["XAI_API_KEY"] = settings.XAI_API_KEY  # or set in your environment

grok_llm = openai_plugin.LLM.with_x_ai(
    model="grok-3-mini",      # pick a supported Grok model
    api_key=settings.XAI_API_KEY
)

session = AgentSession(
    llm=grok_llm,
    stt=assemblyai.STT(),     # streaming STT provider
    tts=cartesia.TTS(),      # TTS provider
    vad=silero.VAD.load(),   # voice activity detection
    preemptive_generation=True,
    max_tool_steps=1,
    user_away_timeout=180,
    min_interruption_duration=0.35,
    min_interruption_words=2,
    min_endpointing_delay=0.3,
    max_endpointing_delay=0.8,
    false_interruption_timeout=1.0,
    turn_detection="vad",
)

What to do if you specifically need low-latency realtime WebSocket voice (Grok) • The websocket realtime endpoint (wss://api.x.ai/v1/realtime) is restricted/not publicly available. Even with a valid XAI key you will get 401 unless xAI granted you realtime access. • If you do have a special realtime access grant from xAI, pass the XAI API key via

api_key=

or

XAI_API_KEY

env var and the underlying client may connect — but most customers cannot use that right now. If you want, tell me: • The model you want to use (grok-3-mini / grok-beta / other) • How you run the app (local, Docker, cloud) I’ll give exact env var placement or a full agent example tuned to your stack. Sources: xAI LLM plugin guide | LiveKit Docs