Hi, can

realtime_audio_output_node()

emit custom audio frames before the first OpenAI Realtime audio frame arrives? I’m trying to implement a non-blocking prerecorded filler/backchannel preroll for avatar lipsync: after the user stops speaking, OpenAI Realtime should start generating the real response immediately; if no model audio is available after ~200–400 ms, I want to emit a short WAV like “mmh / ok / I see” into the same avatar audio stream, then cancel it as soon as the first real model audio frame arrives. The key requirement is that this must not behave like

session.say(...)

or a queued assistant speech item, because that only shifts the latency later. It needs to mask latency while the realtime response is already being generated. Is this supported with

realtime_audio_output_node()

, or does that node only process/transform model audio frames after the model has already started producing them?

Thanks for reaching out, Michele. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Thanks, that clarifies that

realtime_audio_output_node()

cannot emit audio before the first model frame. The remaining key question for our use case is about avatar lipsync. If we publish the filler/backchannel as a separate audio source/track while the OpenAI Realtime response is already being generated, can that separate track still drive the same avatar lipsync/speaking animation? In other words, is there a supported way for a secondary, temporary audio track to be treated as avatar speech for lipsync purposes, without becoming a queued

session.say(...)

item and without blocking the main realtime model response? Desired behavior: 1. User stops speaking. 2. OpenAI Realtime starts generating immediately. 3. A short prerecorded filler WAV is published through a secondary server-side audio source/track. 4. The avatar lipsyncs to that filler. 5. As soon as the first OpenAI Realtime audio frame arrives, the filler track is stopped and the avatar switches to lipsyncing the real model audio. 6. No extra assistant turn is created and no delay is added to the model response. Is this possible with LiveKit’s avatar/audio pipeline today? If yes, should this be implemented as a server-side published LocalAudioTrack, a browser-side WebAudio track, or some other LiveKit pattern?

Acknowledging your question about whether a secondary temporary audio track can drive avatar lipsync without queuing or blocking the realtime model — short answer: yes, but it requires implementing the lipsync pipeline to listen to the active audio track you publish and switching that active source when model audio arrives. LiveKit itself can carry multiple audio tracks simultaneously, but deciding which audio drives an avatar’s lipsync/“speaking” animation is an application-level choice. Recommended approach (concise): 1. Implementation choice ◦ Use a server-side published LocalAudioTrack (or a server-side audio source that publishes into LiveKit). This gives deterministic timing, simpler server-side audio start/stop control, and easier coordination with the model stream. Browser-side WebAudio can work too, but is less reliable for server-driven preroll coordination when the model runs server-side. 1. How to drive lipsync without queuing ◦ Have your avatar/lipsync component subscribe to the specific audio track you publish for speech output (not to session.say items). ◦ When user speech ends: ▪︎ Immediately publish the prerecorded filler on a secondary track and mark it as the “active lipsync source.” ▪︎ Start OpenAI Realtime generation (model runs and begins producing frames as they become available). ◦ Monitor the model audio stream for the first model audio frame. ▪︎ On first model frame arrival: stop/unpublish the filler track (or mute it) and switch the avatar’s active lipsync source to the model audio track. Do not create a queued assistant turn—just switch sources. ◦ Ensure the lipsync component uses live audio amplitude/phoneme extraction from whichever active track is selected so animation switches seamlessly. 1. Important notes ◦

realtime_audio_output_node()

cannot be used to inject preroll; use a separate published track. ◦ Make the filler track low-priority and interruptible; cancel it immediately on the first model frame to avoid overlap or added latency. ◦ Test edge cases: very-short model audio, jittered arrival, and network delays to ensure lipsync handoff is smooth. If you’d like, tell me whether your avatar/lipsync runs in the browser or on the server and which SDK you’re using (JS/TS, native). I’ll provide a short, concrete implementation sketch for that environment. , with interactive elements

Thanks, this is very helpful. In our setup the avatar/lipsync is driven by the audio track published by the LiveKit agent. We now understand that LiveKit can carry a secondary server-side LocalAudioTrack for the filler, but the application/avatar layer must decide which audio track drives lipsync. The key question is: Is there a recommended way, in a LiveKit web avatar setup, to signal the client/avatar component that a temporary secondary audio track should become the active lipsync/speaking source, and then switch back to the main agent/model audio track as soon as the first model audio frame arrives? More specifically: 1. Should the server publish the filler track with a specific track name/metadata, for example

avatar_preroll_filler

? 2. Should the agent send a data message/event to the client such as

active_lipsync_source=filler_track_sid

? 3. When the model audio starts, should the server unpublish/mute the filler track and send another data message such as

active_lipsync_source=agent_audio_track_sid

? 4. Is there any built-in LiveKit convention for this active speech/lipsync source switching, or should we implement it entirely in our avatar client layer? Our goal is to avoid

session.say(...)

, avoid queued assistant turns, and keep OpenAI Realtime generation running in parallel while the avatar lipsyncs to a short filler preroll.

Thanks — I’m responding to your 4 specific questions about signaling a temporary filler track for lipsync and switching back to model audio. Short answers up front: • Yes — publish the filler track with a clear name/metadata. • Yes — send a data/event to the client indicating which track should be the active lipsync source. • Yes — when model audio starts, stop/unpublish or mute the filler and send a data/event switching the active source to the model track. • LiveKit has no built-in convention that automatically switches which track drives avatar lipsync — this must be implemented in your application/avatar client. Recommended pattern (concise, reliable): 1. Track publishing and metadata ◦ Publish the filler as a server-side LocalAudioTrack and set a descriptive track name or metadata (e.g.,

avatar_preroll_filler

) so clients can discover it without guessing SIDs. ◦ Also ensure the agent/model audio track has an identifiable name/metadata (e.g.,

agent_realtime_audio

). 1. Signaling the active lipsync source ◦ Use LiveKit data messages (reliable or unreliable depending on your tolerance) to tell the client which track to use for lipsync. ◦ Message format example (JSON):

{ "type":"active_lipsync", "track_name":"avatar_preroll_filler", "track_sid":"<optional>" }

◦ Have the client resolve the track either by

trackSid

(if provided) or by matching the published track `name`/metadata. 1. Switching back on model audio arrival ◦ On the server, detect the first model audio frame arrival. ◦ Immediately stop/unpublish or mute the filler track (unpublish if you want to remove it; mute if you might reuse it). ◦ Send another data message:

{ "type":"active_lipsync", "track_name":"agent_realtime_audio", "track_sid":"..." }

◦ Client should switch the lipsync input instantly to the model audio track and stop any filler-processing. 1. Client-side lipsync behavior ◦ Implement the lipsync component to accept an explicit active-track selection (by SID or name) rather than always using the first/primary audio track. ◦ When switching sources, smoothly transition lip animation (e.g., zero-crossfade amplitude smoothing) to avoid visual pops. ◦ If the filler is audible and overlapping with model audio for a few ms, prefer muting filler on first model frame to avoid double-speech. 1. Delivery and reliability considerations ◦ Use a reliable data message for the initial active-source switch to avoid missed signals. ◦ Still make the client robust: if a data message is missed, allow the client to fallback by choosing the louder live track or by listening for model frames on known agent track. ◦ Keep filler short and interruptible; ensure server-side timing cancels it immediately on first model frame. Why this works • LiveKit can carry multiple simultaneous audio tracks, but which one drives your avatar is an application decision. Explicit signaling plus clear track metadata gives deterministic, low-latency control without queuing a session.say turn. If you want, I can provide: • A small example message schema and client-side pseudocode for selecting tracks and switching lipsync in a LiveKit web (JS) avatar. Which would you prefer: pseudocode for the browser lipsync component or a short server-side sequence for publishing/muting the filler track?, with interactive elements