https://livekit.io logo
Join Slack
Powered by
The Realtime openai agent has input_audio_transcri...
# ask-ai
a
The Realtime openai agent has input_audio_transcription as input param where I can change the transcription model. However this only allows me to choose openai transcription model. Is it possible to use other model (such as speechmatic, deepgram etc) with the realtime open ai agent - but only for transcriptions?
t
Thanks for reaching out, Malte. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’m answering your question about using third‑party STT (Deepgram, Speechmatics, etc.) with the Realtime OpenAI agent’s transcription. Short answer: Yes — but not via the 
input_audio_transcription
parameter. That parameter only accepts OpenAI’s transcription model. To use a different STT provider, add a separate STT plugin to your AgentSession so the agent runs OpenAI Realtime for the agent LLM/audio generation while a third‑party STT handles transcription in parallel. What that looks like and why it helps • 
input_audio_transcription
is limited to OpenAI’s transcription model. • You can configure an 
stt
plugin (e.g., Deepgram, Speechmatics) in your 
AgentSession
. This gives you real‑time/interim transcripts and enables using the third‑party provider for turn detection or live captions while the OpenAI Realtime model handles generation and agent behavior. • This runs both pieces side‑by‑side (OpenAI Realtime for the agent, third‑party STT for transcription). It does incur additional cost from the external STT provider. If you’d like, I can: • Provide a short example of how to configure your 
AgentSession
with a specific STT plugin (tell me which provider you want: Deepgram, Speechmatics, etc.). • Explain how to use the STT output for turn detection vs. using server-side turn detection. • Walk through handling interim transcripts and post‑turn processing for your integration. Which provider would you like an example for, and do you want the STT to handle endpointing (turn detection) or only produce transcripts? Sources: OpenAI Realtime API plugin guide | LiveKit Documentation | Realtime models overview | LiveKit Documentation | Turns overview | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation
a
sure give me an example
t
Thanks — I’ll show a concise example of configuring an 
AgentSession
to use Deepgram as the STT provider alongside the OpenAI Realtime agent. This keeps OpenAI Realtime for the agent/LLM audio generation while Deepgram provides real‑time transcription (including interim results) and can optionally be used for turn detection. Below are two examples: • A short Python example showing how to wire Deepgram into 
AgentSession
. • A short JavaScript/TypeScript example for a browser/node client. If you prefer Speechmatics instead of Deepgram, tell me and I’ll adapt the snippet. Python example
Copy code
# Pseudocode / illustrative example
from openai.realtime import RealtimeModel, AgentSession
from deepgram import DeepgramSTT  # hypothetical SDK wrapper

# Initialize models / clients
openai_llm = RealtimeModel(model="gpt-realtime-1")  # Realtime agent model
deepgram_stt = DeepgramSTT(api_key="DEEPGRAM_API_KEY", model="general")  # provider-specific config

# Create AgentSession with both LLM and STT configured
session = AgentSession(
    llm=openai_llm,
    stt=deepgram_stt,
    # Optional: choose turn detection source: "stt" to use Deepgram endpointing,
    # or "server" / "llm" to use OpenAI server-side turn detection.
    turn_detection="stt"
)

# Start the session and route audio
# - Microphone audio → sent to Deepgram for real-time transcription.
# - Audio is also sent to OpenAI Realtime for agent audio generation.
session.start()

# Example handlers:
def on_transcript(partial, final):
    if partial:
        print("Interim:", partial)
    if final:
        print("Final:", final)
        # forward final transcript to agent if needed or handle separately

session.on("transcript", on_transcript)

# When done:
session.close()
JavaScript / TypeScript example
Copy code
// Pseudocode / illustrative example
import { RealtimeModel, AgentSession } from "openai/realtime";
import { DeepgramSTT } from "deepgram-stt-sdk"; // hypothetical SDK

const openaiLLM = new RealtimeModel({ model: "gpt-realtime-1" });
const deepgram = new DeepgramSTT({ apiKey: process.env.DG_API_KEY, model: "general" });

const session = new AgentSession({
  llm: openaiLLM,
  stt: deepgram,
  turn_detection: "stt", // use Deepgram for endpointing
});

session.on("stt.partial", (txt) => {
  console.log("Interim:", txt);
});

session.on("stt.final", (txt) => {
  console.log("Final:", txt);
  // you can send the final transcript into the agent pipeline if desired
});

await session.start();

// route microphone / audio tracks into session
// when finished:
await session.close();
Key notes and options • The 
input_audio_transcription
parameter accepts only OpenAI’s transcription model. Use the 
stt
property on 
AgentSession
to attach a third‑party STT. • Using a third‑party STT gives you real‑time/interim transcripts and can be used for turn detection by setting 
turn_detection
to 
"stt"
. • You’ll incur separate billing from the STT provider. • Decide whether you want the STT transcripts forwarded into the agent as text input (common) or just used for captions/analytics and let OpenAI handle the agent turn/response logic. Would you like: • A concrete, copy‑pasteable example for a specific runtime (browser or Node) and provider (Deepgram or Speechmatics)? • Guidance on handling interim vs final transcripts when forwarding to the agent?
2 Views
Open in Slack
PreviousNext
Powered by