https://livekit.io logo
Join Slack
Powered by
how Using GPT-4o mini Realtime in livket agent se...
# ask-ai
b
how Using GPT-4o mini Realtime in livket agent server ts?
✅ 1
t
Thanks for reaching out, Alejo. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m responding to your question about using GPT‑4o Mini Realtime (OpenAI Realtime) in a LiveKit agent server written in TypeScript. Below is a concise, working approach and example you can follow. If you want something tailored to your project (existing repo, voice choices, TTS setup), tell me more about your current setup and I’ll adjust the steps. Prerequisites • Install the OpenAI plugin for LiveKit agents:
Copy code
pnpm add "@livekit/agents-plugin-openai@1.x"
• Set your 
OPENAI_API_KEY
in your environment (e.g., 
.env
). Basic TypeScript example 1. Create an AgentServer and use 
openai.realtime.RealtimeModel
as the session LLM. The 
RealtimeModel
defaults to the 
gpt-realtime
model (GPT‑4o Mini Realtime).
Copy code
import { voice, AgentServer, JobContext } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

const server = new AgentServer();

server.rtcSession({ agentName: 'my-agent' })(
  async (ctx: JobContext) => {
    const session = new voice.AgentSession({
      llm: new openai.realtime.RealtimeModel({
        voice: 'alloy',       // choose from available voices (e.g., 'alloy', 'shimmer', 'nova')
        temperature: 0.8,     // controls randomness
        // modalities: ['text','audio'], // default; set to ['text'] to disable audio streaming TTS
        // model: 'gpt-realtime' // optional explicit model ID (defaults to gpt-realtime)
      }),
    });

    await session.start({ room: ctx.room, agent: new voice.Agent() });
  }
);

// Run with your normal start script (e.g., pnpm dev)
Key configuration notes • voice: pick a voice like 
'alloy'
, 
'shimmer'
, or 
'nova'
for audio output. • temperature: controls response randomness (commonly set 0.6–1.2). • modalities: default is 
['text','audio']
. Use 
['text']
if you want to handle TTS separately. • turn_detection: optional; you can enable OpenAI’s VAD for automatic voice turn detection. When to disable audio modality • If you prefer to run separate TTS (e.g., external TTS service or later-stage audio generation), set 
modalities: ['text']
and use your own TTS pipeline. What I can help next • Add sample code to handle incoming audio streams or user utterances in the LiveKit room. • Show how to switch to text-only mode and connect a separate TTS. • Help choose voice and 
turn_detection
settings for your use case. • Troubleshoot errors if you paste the code or runtime logs you’re seeing. Would you like a version that includes event handlers for incoming audio/user messages or an example that uses text-only + external TTS? Sources: OpenAI Realtime API plugin guide | LiveKit Documentation, with interactive elements
b
and gpt-realtime-1.5
t
Thanks — I’m following up on your question about using the gpt-realtime-1.5 model in a LiveKit agent server (TypeScript). Do you want: • a code example showing how to explicitly set 
gpt-realtime-1.5
instead of the default 
gpt-realtime
? • details on how 
gpt-realtime-1.5
differs from the default 
gpt-realtime
for LiveKit usage (modality support, latency, voice/TTS behavior)? • guidance on configuration values (temperature, modalities, turn detection) that work best with 
gpt-realtime-1.5
? • help migrating an existing AgentSession that currently uses 
gpt-realtime
to 
gpt-realtime-1.5
? If you want the explicit code change, here’s the minimal adjustment you can make in the TypeScript example I shared earlier (I’ll expand with a full snippet if you want):
Copy code
llm: new openai.realtime.RealtimeModel({
  model: 'gpt-realtime-1.5',
  voice: 'alloy',
  temperature: 0.8,
})
Which of the four options above should I expand on (or do you have a different question about gpt-realtime-1.5)? , with interactive elements
b
and gpt-realtime-1.5?
t
Thanks — I’m responding to your follow-up about using the gpt‑realtime‑1.5 model in a LiveKit agent server (TypeScript). Yes — you can explicitly use 
gpt-realtime-1.5
in the same way as the default realtime model. Below is a concise TypeScript example and a few actionable configuration notes to help you get started. 1) Minimal TypeScript example (explicit model)
Copy code
import { voice, AgentServer, JobContext } from '@livekit/agents';
import * as openai from '@livekit/agents-plugin-openai';

const server = new AgentServer();

server.rtcSession({ agentName: 'my-agent' })(
  async (ctx: JobContext) => {
    const session = new voice.AgentSession({
      llm: new openai.realtime.RealtimeModel({
        model: 'gpt-realtime-1.5', // explicitly select gpt-realtime-1.5
        voice: 'alloy',            // choose voice: 'alloy', 'shimmer', 'nova', etc.
        temperature: 0.8,          // adjust randomness
        // modalities: ['text','audio'], // default; set to ['text'] if you want external TTS
        // turn_detection: { type: 'semantic' } // optional VAD config
      }),
    });

    await session.start({ room: ctx.room, agent: new voice.Agent() });
  }
);

// Run normally (e.g., pnpm dev)
2) Practical configuration tips • model: Set 
model: 'gpt-realtime-1.5'
to explicitly use that variant. • voice: Pick a voice that fits your use case (e.g., 
'alloy'
, 
'shimmer'
, 
'nova'
). • temperature: Use ~0.6–1.0 for conversational responses; raise for more creative output. • modalities: Keep 
['text','audio']
for built-in streaming speech. Use 
['text']
when you want to generate text and handle TTS separately. • turn_detection: Enable OpenAI’s VAD if you want automatic voice turn handling (semantic or server VAD options). 3) When to choose gpt-realtime-1.5 • Use 
gpt-realtime-1.5
when you want that specific realtime model variant explicitly. It integrates the same way as the default realtime model in LiveKit’s RealtimeModel wrapper, so no special plumbing is required beyond setting the 
model
field. 4) Next steps I can help with • Add event handlers for incoming audio or text messages in the LiveKit room. • Show a version using text-only + an external TTS pipeline. • Tune 
turn_detection
and voice selection for better UX in your scenario. • Help troubleshoot any runtime errors if you paste logs or code. Which would you like next — a full example with message handlers, or a text-only + external TTS example? , with interactive elements
6 Views
Open in Slack
PreviousNext
Powered by