https://livekit.io logo
Join Slack
Powered by
when using the openai realtime api, is it possible...
# ask-ai
w
when using the openai realtime api, is it possible to make the llm output text, and then pipe that text into a tts model?
t
Thanks for reaching out, Drew. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — that’s a great question. Yes: you can configure the OpenAI Realtime API to output text and then pipe that text into a TTS model using LiveKit. How it works (high level) • Configure the Realtime LLM to produce text only by setting its 
modalities
to 
["text"]
. This makes the model return text responses instead of audio. • Take the text output and pass it to whatever TTS model or provider you want. LiveKit supports wiring that together inside an 
AgentSession
, so the LLM handles understanding/generation and your chosen TTS handles synthesis. Quick examples 1) Python
Copy code
session = AgentSession(
    llm=openai.realtime.RealtimeModel(modalities=["text"]),
    tts="cartesia/sonic-3"  # or another TTS model/provider
)
2) Node.js / TypeScript
Copy code
const session = new voice.AgentSession({
    llm: new openai.realtime.RealtimeModel({
        modalities: ["text"]
    }),
    tts: "cartesia/sonic-3",
});
Notes and tips • This pattern is often called a “half-cascade” or “half-duplex” setup: the Realtime API handles text generation (and speech comprehension if you also capture audio input), and you keep full control over the speech synthesis step. • The same approach works with Azure OpenAI’s realtime plugin. • If you plan to load conversation history, the Realtime API may default to text output — explicitly setting 
modalities: ["text"]
ensures the behavior you want. If you’d like, tell me: • which TTS provider or model you want to use, and • whether you need the final audio streamed back into a LiveKit room or saved as files— and I’ll give a focused example for that scenario. Sources: OpenAI Realtime API plugin guide | LiveKit Documentation | Realtime models overview | LiveKit Documentation | Azure OpenAI Realtime API and LiveKit | LiveKit Documentation
w
Can I do a similar thing with gemini 2.0 flash?
t
Thanks — good follow-up. Short answer: no, not in the same way. What’s different • Gemini 2.0 Flash is a standard (non-realtime) LLM that only accepts text input and returns text output. It does not support realtime audio I/O or response modalities, so you cannot configure it like a realtime model to emit text while also handling audio directly. • That means you must use a classic pipeline: STT → Gemini 2.0 Flash (text in/out) → TTS. Alternative that does support the “half-cascade” pattern • Use the Gemini Live API (the realtime plugin). Gemini Live supports configurable output modalities (text or audio). You can set it to text-only (e.g., 
modalities=["TEXT"]
) so the model produces text which you then send to your TTS of choice — the exact same half-cascade/half-duplex approach you described for OpenAI Realtime. Additional note • Gemini 2.0 Flash is deprecated and scheduled for retirement (February 13, 2026). Consider migrating to Gemini 2.5 Flash / Flash Lite or using Gemini Live if you need realtime-modal behavior. If you want, tell me: • whether you prefer to keep using Gemini 2.x (and which version), or • which TTS provider you plan to use, and whether you need streamed audio back into a LiveKit room or saved files — and I’ll give a focused example or migration recommendation. Sources: Realtime models overview | LiveKit Documentation | Gemini Live API plugin | LiveKit Documentation | Large language models (LLM) overview | LiveKit Documentation | Google Gemini LLM | LiveKit Documentation
w
does livekit support personaplex?
t
Thanks — good question. I checked LiveKit’s docs and integration notes: LiveKit does not mention support for Personaplex, and there’s no reference to a Personaplex plugin or integration in the sources I can access. If you’re exploring an integration, two quick ways I can help next: • If you tell me what you want Personaplex to do with LiveKit (TTS, avatar control, conversational agent, streaming audio, etc.), I can suggest equivalent supported options or a recommended integration pattern. • If you have Personaplex’s API details (authentication method, streaming vs. REST TTS endpoints), share those and I can sketch how to connect it to LiveKit (for example, piping text output into Personaplex TTS, or routing Personaplex audio into a LiveKit room). Which direction would you like to take? Sources: An open source stack for real-time multimodal AI | Perplexity LLM plugin guide | LiveKit Documentation | examples/avatar_agents | livekit/livekit_composite | DeepWiki | Tavus virtual avatar integration guide | LiveKit Documentation
w
what about eleven v3
t
Thanks — good question. Short answer: Eleven V3 is not listed as a supported model in LiveKit’s docs or pricing info. What LiveKit does support • The documented ElevenLabs TTS models available through LiveKit are: ◦ 
elevenlabs/eleven_flash_v2
◦ 
elevenlabs/eleven_flash_v2_5
◦ 
elevenlabs/eleven_turbo_v2
◦ 
elevenlabs/eleven_turbo_v2_5
◦ 
elevenlabs/eleven_multilingual_v2
Notes • LiveKit’s ElevenLabs plugin defaults to 
eleven_flash_v2_5
in examples, and 
eleven_multilingual_v2
is shown for multilingual use. • Pricing and LiveKit Inference lists the V2 and v2.5 models; there’s no mention of an Eleven V3. If you’d like, I can: • Show a snippet for wiring one of the supported ElevenLabs models into an AgentSession or pipeline. • Help you pick which Eleven model fits your needs (e.g., best multilingual support vs. naturalness). • Sketch how to route ElevenLabs audio back into a LiveKit room or save it as files. Which would you prefer? Sources: ElevenLabs TTS | LiveKit Documentation | Build voice, video, and physical AI | LiveKit | ElevenLabs TTS plugin guide | LiveKit Documentation | Pipeline Translator Agent | LiveKit Documentation
Open in Slack
PreviousNext
Powered by