Question how do you make a Realtime agent speak a configured LiveKit Community #ask-ai

Question: how do you make a Realtime agent speak a...

careful-monitor-57946

05/19/2026, 11:15 PM

Question: how do you make a Realtime agent speak a configured greeting verbatim at the start of a phone call? Setup: -

@livekit/agents@1.4.2

@livekit/agents-plugin-openai@1.4.2

- Single

voice.AgentSession({ llm: new openai.realtime.RealtimeModel({ model: 'gpt-realtime', voice: 'marin', ... }) })

— no separate STT/TTS (Realtime model is the whole pipeline) - SIP inbound calls via LiveKit - Each company has a configured first-line greeting (e.g.

"Hi, thank you for calling Chucky Cheese, this is Tony — you are on a recorded line. How can I help?"

) that we need spoken verbatim when the call connects, before any user input. What I've tried: 1.

session.say(greeting, { allowInterruptions: true })

→ throws

"trying to generate speech from text without a TTS model"

because we don't configure a separate TTS pipeline (the realtime model owns audio). 2.

session.generateReply({ instructions: greeting })

→ the model hallucinates a different greeting (e.g. says "Welcome to fun zone" for the Chucky Cheese prompt).

instructions

is a system-prompt-style directive for the turn, not text to speak. 3. `session.generateReply({ instructions:

With an upbeat tone, immediately say "${greeting}". Do not say it multiple times.

})` → mostly works, but the model occasionally leaks fragments of the wrapper meta-text (e.g. callers hear "with an upbeat attitude…" before the actual greeting). Security/UX issue for us since we're customer-facing. Is there a canonical pattern for "Realtime agent speaks this exact text first, then proceeds normally"? I've thought about: - Pre-populating the

Agent.chatCtx

with a synthetic

ChatMessage({ role: 'assistant', content: greeting })

and then calling

session.generateReply()

with no args — but I'm not sure the realtime model will actually speak a context-only item. - Mounting a separate

tts:

on the session purely so

session.say()

works — feels wasteful since the realtime model already does audio. - Adding

voice.Agent.onEnter()

override — but unclear what the right hook is for "speak this verbatim." Is there a method on

voice.AgentSession

realtime.RealtimeModel

I'm missing for "play this exact text through the realtime model's audio" without going through the LLM-instruction path? Pointers to a sample repo would be great.

✅ 1

tall-airline-16756

05/19/2026, 11:15 PM

Thanks for reaching out, fideki5170. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

05/19/2026, 11:15 PM

Thanks — I’m responding to your question about making a Realtime agent speak a configured greeting verbatim at call start. Short answer: there is no method on

voice.AgentSession

realtime.RealtimeModel

that will reliably play an exact script verbatim through the Realtime model’s audio pipeline without going through an LLM-generation step. The Realtime model treats text as instructions/generation input, so

generateReply()

can paraphrase or leak prompt text, and

session.say()

requires a separate TTS pipeline. Recommended (reliable) option — half‑cascade (what I suggest) 1. Use the Realtime model only for the LLM (set it to return text responses), and add a separate TTS instance for audio. 2. Then call

session.say(greeting, { allowInterruptions: true })

to play the greeting verbatim at connect. ◦ This guarantees exact playback and avoids hallucination or meta-text leakage. ◦ Example pattern:

Copy code

const session = new voice.AgentSession({
  llm: new openai.realtime.RealtimeModel({ model: 'gpt-realtime', modalities: ['text'], /* voice removed */ }),
  tts: 'your-tts-model-id' // enable TTS so session.say() works
});

await session.say(greeting, { allowInterruptions: true });
await session.generateReply(); // continue normal turn handling

Why other approaches are unreliable •

session.say(greeting)

with only the Realtime model fails with “trying to generate speech from text without a TTS model.” •

generateReply({ instructions: greeting })

treats the greeting as a system instruction and often paraphrases or hallucinates. • Wrapping the greeting inside an instruction to “immediately say …” can leak the wrapper text to the caller. • Pre-loading assistant messages into

Agent.chatCtx

is not guaranteed to cause the Realtime model to speak that context verbatim. If you must keep a single Realtime model (no TTS) • You can try very minimal, exact-prompting (no wrapper text) but this is not guaranteed to be verbatim and may still leak or paraphrase. There is no documented supported API to force verbatim playback in the pure Realtime pipeline. If you want, I can help with: • Configuring the half‑cascade (example TTS options and exact session setup). • A minimal fail-safe flow that plays pre-recorded audio (if you prefer prerecorded greetings rather than TTS). Which would you like me to provide next? Sources: Agent speech and audio | LiveKit Documentation | Realtime models overview | LiveKit Documentation, with interactive elements

3 Views

Open in Slack

Previous Next