hey

Thanks for reaching out, Kushank. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I have multiple participant in the agentsession but no audio is coming. Multiple participants consists of Avatar providers

Thanks — I’m looking at your report that multiple participants in the AgentSession are avatar providers but no audio is coming through. I can help troubleshoot — a few quick checks to narrow this down: 1. Do you see the avatar worker participants in the room (a separate participant for each avatar provider)? 2. Are you using an avatar plugin (Anam, D‑ID, Tavus, etc.) and have their API keys configured? 3. Does your frontend subscribe to the avatar worker’s audio track (not just the agent’s track)? If unsure, share which SDK/platform (Web, iOS, Android) you’re using. 4. What are your RoomOptions for audio output — is

audio_output

disabled? 5. Any relevant logs: do you see "registered worker" or errors when starting the avatar session? Immediate steps you can try: • Confirm the avatar worker was registered and appears in participants. Play that participant’s audio track. • If you publish agent TTS, ensure the avatar worker received the audio and republished it (agent should not also publish audio directly when using avatars). • If you’re on Web, try the

useVoiceAssistant

hook to automatically select the avatar audio track. Tell me the answers to 1–5 and which SDK/platform you’re on and I’ll provide the exact checks or code snippet to validate/subscribed tracks. Sources: Video | LiveKit Documentation | Text and transcriptions | LiveKit Documentation | Virtual avatar models overview | LiveKit Documentation | D-ID Realtime Avatar integration guide | LiveKit Documentation | Anam virtual avatar integration guide | LiveKit Documentation, with interactive elements

Setup: LiveKit Agents 1.2.14 + livekit-plugins-bey (Beyond Presence). We pre-warm multiple bey avatars in the same room — one per chapter — and switch session.output.audio between their sinks at chapter handoff. To avoid identity collision (bey defaults all workers to bey-avatar-agent), we pass a unique avatar_participant_identity per worker (e.g. bey-avatar-2bc759ab-a7e). Frontend is Web (Next.js 15, livekit-client, @strong-father-34524/components-react). --- 1. Avatar worker participants in the room? Yes — confirmed in logs and on the frontend. Each bey worker joins with its custom identity: DEBUG livekit.agents waiting for the remote participant {"identity": "bey-avatar-2bc759ab-a7e"} DEBUG livekit.agents remote participant ready {"identity": "bey-avatar-2bc759ab-a7e"} INFO agent [Avatar] ✅ Video track ready from bey-avatar-2bc759ab-a7e (waited 4.5s) Frontend renders that participant's video track correctly (avatar tile is visible). 2. Avatar plugin + API keys? livekit-plugins-bey (Beyond Presence). BEY_API_KEY is set; avatar.start() succeeds for every worker — no API errors. We instantiate as: bey.AvatarSession( avatar_id="<uuid>", avatar_participant_identity="bey-avatar-2bc759ab-a7e", # custom per-chapter ) 3. Frontend audio subscription? Web SDK. We don't filter by identity — VolumeControlledAudioRenderer subscribes to all remote audio tracks: room.on('trackSubscribed', (track, _pub, participant) => { if (track.kind === 'audio' && track instanceof RemoteAudioTrack) { track.attach(); // also iterates existing participant.audioTrackPublications on mount } }); Auto-subscribe is enabled (default). Video tracks from the bey worker DO get subscribed and rendered. But the bey worker's audio track never appears — participant.audioTrackPublications is empty for bey-avatar-2bc759ab-a7e. 4. RoomOptions for audio output? Default RoomOutputOptions (audio_output enabled). Agent's session.output.audio is replaced with bey's DataStreamAudioOutput (set inside bey.AvatarSession.start()). The expected log fires: WARNI livekit.agents RoomIO audio output is enabled but output.audio is already set, ignoring.. So the agent isn't double-publishing audio — all TTS routes through bey. 5. Worker registration / errors? Worker registers cleanly, and bey acknowledges it received and played the agent's TTS: INFO agent [Avatar] Routed session.output.audio to identity=bey-avatar-2bc759ab-a7e INFO agent [Avatar] ✅ avatar_changed published: None → bey-avatar-2bc759ab-a7e (force=True) INFO livekit.agents TTS metrics audio_duration: 8.83s INFO livekit.agents playback finished event received {"caller_identity": "bey-avatar-2bc759ab-a7e"} No errors. Bey's lk.playback_finished RPC works in both directions, so the data path agent ↔️ worker is healthy. --- The puzzle: Bey signals playback_finished for the audio we forwarded, but the bey worker is not republishing it as an audio track on its participant (only video). This is consistent across every chapter — including the very first one with only one bey worker in the room. The bey plugin source publishes the worker token with with_kind("agent") + ATTRIBUTE_PUBLISH_ON_BEHALF=<our agent identity>. We're wondering whether publish_on_behalf causes the LiveKit server to suppress the bey worker's published audio track when subscribers are not the on-behalf-of identity, or whether bey's worker-side service simply doesn't republish audio when given a custom avatar_participant_identity (regression?). The exact thing we'd love confirmed: With avatar_participant_identity set on bey.AvatarSession(...), does the bey worker still publish a microphone audio track on its participant, and is that track visible/subscribable to a regular (non-agent) Web client connected to the same room? If not, what's the supported way to get bey-lipsynced audio to a plain Web viewer?

Thanks — this is really clear and helpful context. Short answer to your exact question: • Setting

avatar_participant_identity

does not inherently prevent the Bey worker from publishing an audio track. The Bey worker should publish lip‑synced audio and video on its participant, and that audio should be visible/subscribable to a regular Web client. Why you’re seeing only video (likely causes): • The Bey worker may not actually be publishing the audio despite receiving TTS (possible regression in the Bey service). • The frontend subscription/attachment logic may be missing the audio track due to track source/type filtering or a timing/race condition (audio published after your initial iteration). • The audio may be published with a non‑MICROPHONE source tag that your playback logic ignores. Recommended next steps (quick, ordered): 1. Add logging in your

trackSubscribed

handler to print

track.kind

,

track.source

, and

participant.identity

to confirm whether an audio track ever arrives. 2. Inspect

room.participants

for the bey worker and log

participant.audioTrackPublications

periodically (or on

participantUpdated

) to catch late-published tracks. 3. Try the

useVoiceAssistant()

hook from

@livekit/components-react

(it handles

lk.publish_on_behalf

and avatar worker pairing automatically). 4. As a quick repro, test with the default

avatar_participant_identity

(no custom identity) or a single warmed avatar in an otherwise empty room to see if audio appears — if it does, that points to a Bey regression when using custom identities or pre-warming. 5. Log Bey plugin/service-side traces around publication (confirm worker actually calls LiveKit Publish for an audio track after playback_finished). If you’d like, share: • a short snippet of your

trackSubscribed

logs (with the extra fields above), or • results from the default-identity single-avatar test — and I’ll help interpret the logs and give the exact code tweak to ensure the audio attaches for regular Web viewers. Sources: Understand and Improve Agent Latency | LiveKit | Beyond Presence virtual avatar integration guide | LiveKit Documentation | Video | LiveKit Documentation | Bringing AI avatars to voice agents | LiveKit | Virtual avatar models overview | LiveKit Documentation, with interactive elements

Apr 28, 025728.479 AM livekit.agents RoomInputOptions and RoomOutputOptions are deprecated, use RoomOptions instead Apr 28, 025728.479 AM livekit.agents RoomIO audio output is enabled but output.audio is already set, ignoring.. Apr 28, 025728.481 AM livekit.agents input stream attached Apr 28, 025728.481 AM livekit.agents input stream detached Apr 28, 025728.481 AM livekit.agents input stream attached Apr 28, 025728.535 AM agent [UnifiedAgent #1] on_enter called for chapter 1 Apr 28, 025728.537 AM livekit.agents using audio io:

RoomIO

->

RecorderIO

->

AgentSession

->

RecorderIO

->

TranscriptSynchronizer

->

DataStreamIO

Apr 28, 025728.537 AM livekit.agents resume_false_interruption is enabled but audio output does not support pause, it will be ignored Apr 28, 025728.537 AM livekit.agents using transcript io:

AgentSession

->

TranscriptSynchronizer

->

RoomIO

Apr 28, 025728.537 AM agent [Initialization] ✅ Session started successfully with chapter agent Apr 28, 025728.537 AM agent [Suggestions] Cleared suggestions from frontend Apr 28, 025728.537 AM agent [UnifiedAgent #1] Video data found on attempt 1 Apr 28, 025728.538 AM agent [Langfuse] Updated current observation: set_chapter Apr 28, 025728.540 AM agent [UnifiedAgent #1] Set chapter to 1 Apr 28, 025728.540 AM agent [UnifiedAgent #1] Skipping auto-start (onboarding not completed), injecting chapter context Apr 28, 025728.543 AM agent [Langfuse] Updated current trace: get_chapter_teaching_instructions Apr 28, 025728.543 AM agent [ChapterContentService] Available chapters: [1, 2, 3, 4, 5, 6] Apr 28, 025728.543 AM agent [ChapterContentService] Requested Chapter 1 (type: <class 'int'>) Apr 28, 025728.543 AM agent [ChapterContentService] Chapter 1 exists: True Apr 28, 025728.543 AM agent [ChapterContentService] 🔍 Checking Chapter 1 Apr 28, 025728.543 AM agent [ChapterContentService] 🔍 Available chapters: [1, 2, 3, 4, 5, 6]... (showing first 10) Apr 28, 025728.543 AM agent [ChapterContentService] 🔍 Total chapters: 6 Apr 28, 025728.543 AM agent [ChapterContentService] 🔍 Chapter 1 exists: True Apr 28, 025728.547 AM agent [UnifiedAgent #1] Injected chapter 1 teaching instructions into chat context Apr 28, 025729.565 AM livekit.plugins.deepgram Established new Deepgram STT WebSocket connection: Apr 28, 025731.116 AM opentelemetry.exporter.otlp.proto.http.trace_exporter Failed to export span batch code: 401, reason: {"message":"Invalid credentials. Confirm that you've configured the correct host."} Apr 28, 025733.494 AM agent [SpeechCreated] Speech started for chapter 1 - clearing suggestions Apr 28, 025733.494 AM agent [SpeechCreated] Speech started for chapter 1 (speech_id: 5629156896) Apr 28, 025733.495 AM agent [SpeechMonitor] Waiting for speech to complete (chapter 1) Apr 28, 025733.506 AM agent [Suggestions] Cleared suggestions from frontend Apr 28, 025733.539 AM agent [DataHandler] Checking for unlogged assistant messages before logging user message Apr 28, 025733.539 AM agent [DataHandler] session.history exists Apr 28, 025733.540 AM agent [DataHandler] Found 1 items in history Apr 28, 025733.540 AM agent [DataHandler] No assistant message found in history Apr 28, 025733.540 AM agent [DataHandler] Logging user text message from Kushank: Hello Apr 28, 025733.542 AM agent [ConversationItemAdded] User message (5 chars): Hello Apr 28, 025733.545 AM session_persistence [SessionPersistence] Incremental save: 1 messages Apr 28, 025733.561 AM agent [Suggestions] Cleared suggestions from frontend Apr 28, 025734.287 AM session_persistence [SessionPersistence] Saved progress: chapter=1, completed=[], covered=[1] Apr 28, 025734.325 AM opentelemetry.exporter.otlp.proto.http.trace_exporter Failed to export span batch code: 401, reason: {"message":"Invalid credentials. Confirm that you've configured the correct host."} Apr 28, 025734.543 AM agent [SpeechCreated] Speech started for chapter 1 - clearing suggestions Apr 28, 025734.543 AM agent [SpeechCreated] Speech started for chapter 1 (speech_id: 5587742512) Apr 28, 025734.544 AM agent [SpeechMonitor] Waiting for speech to complete (chapter 1) Apr 28, 025734.544 AM agent [DataHandler] Checking for unlogged assistant messages before logging user message Apr 28, 025734.545 AM agent [DataHandler] session.history exists Apr 28, 025734.545 AM agent [DataHandler] Found 2 items in history Apr 28, 025734.545 AM agent [DataHandler] No assistant message found in history Apr 28, 025734.545 AM agent [DataHandler] Logging user text message from Kushank: Hello Apr 28, 025734.553 AM agent [Suggestions] Cleared suggestions from frontend Apr 28, 025734.557 AM agent [ConversationItemAdded] User message (5 chars): Hello Apr 28, 025734.564 AM agent [Suggestions] Cleared suggestions from frontend Apr 28, 025734.570 AM agent [SpeechMonitor] Speech interrupted by user - skipping callbacks Apr 28, 025736.331 AM livekit.agents LLM metrics Apr 28, 025736.332 AM agent [UnifiedAgent] Generation completed for chapter 1 Apr 28, 025736.332 AM agent [LLM] Buffered assistant text ready (len=88, last_query_type='course', current_chapter=1)

Thanks — this is very thorough. I’ve reviewed your logs and setup and here’s a concise diagnosis and next steps. What I can confirm • Setting

avatar_participant_identity

does NOT prevent the Bey worker from publishing audio. The Bey worker should republish lip‑synced audio + video as a normal participant, and that audio is subscribable by regular Web clients. • Your logs show the Agent → Bey data path is healthy (playback_finished, TTS duration), and RoomIO warnings are expected when the avatar session overrides

output.audio

. Most likely causes for seeing video but no audio • The Bey worker is not actually publishing the audio track (possible Bey-side regression). • Frontend subscription/attachment logic is missing the audio track due to source/type filtering or a timing/race condition (audio published after your initial checks). • The audio track is published with an unexpected

track.source

that your playback code ignores. Concrete checks to run now (ordered) 1. Enhance your

trackSubscribed

logs to print: ◦

track.kind

,

track.source

,

participant.identity

, and

pub.trackSid

◦ Also log in

participantUpdated

the full

participant.audioTrackPublications

. 1. Poll

room.participants.get('bey-avatar-...')

a few seconds after playback_finished and log

audioTrackPublications

(to catch late publishes). 2. In

trackSubscribed

, do not filter by

source

— simply

track.attach()

for any

audio

kind to confirm arrival. 3. Try

useVoiceAssistant()

from

@livekit/components-react

to let it handle

lk.publish_on_behalf

pairing automatically and see if audio appears. 4. Quick repro: run a single avatar (no pre-warm) with default identity — if audio appears there but not with custom identities/pre-warming, that points to a Bey regression in your multi‑avatar flow. What to log / paste if you want me to dig deeper • The enhanced

trackSubscribed

/

participantUpdated

output around the time you see

playback_finished

. • The

participant

object for the bey worker (including attributes). • A minimal repro steps (single avatar default identity vs your pre-warm flow) and results. If you share those logs I’ll pinpoint whether it’s a client-side subscription issue or a Bey publish regression and give the exact code tweak or workaround. , with interactive elements

but we can have multiple avatar provider pre warmed as participants and make them speak one at a time?

Yes — you can pre‑warm multiple Bey avatar workers as participants in the same room and have them speak one at a time. How to do it (high level) • Create one worker per avatar and give each a unique

avatar_participant_identity

(you already do this). • Route the agent’s TTS to the active avatar by switching the session’s audio output to that avatar’s sink (what you’re doing with

session.output.audio

). Only the active worker should receive and republish audio. • Ensure workers are started (registered) ahead of time so they’re ready to accept TTS when you switch chapters. Key operational details and best practices • Avoid having the agent itself publish audio when using avatars; TTS should go to the avatar worker which republishes lip‑synced audio+video. • When switching who speaks, update the agent → avatar routing (you already log the routing); wait for the avatar to acknowledge playback readiness before signaling clients to listen. • On the frontend, subscribe to the avatar worker’s audio track (identify the worker by

participant.identity

and/or

attributes['lk.publish_on_behalf'] === agentIdentity

). Use

useVoiceAssistant()

to simplify this pairing. • Watch for timing/race issues: audio may be published slightly after video — poll

participant.audioTrackPublications

or listen for `trackSubscribed`/`participantUpdated` events after you switch sinks. Troubleshooting tips • If video shows but audio doesn’t for a warmed worker, verify the Bey worker actually published an audio track after playback (log

track.kind

,

track.source

,

participant.identity

, and

pub.trackSid

). • Test a minimal repro: single avatar (no pre‑warm) vs pre‑warmed multiple avatars — if audio appears only in single-avatar runs, that points to a Bey-side regression in the pre‑warm flow. If you want, tell me: • Whether you need an example snippet to switch

session.output.audio

safely between workers, or • The logs from a switch (playback_finished plus frontend `trackSubscribed`/`participantUpdated`) and I’ll give exact code and checks to ensure the speaking avatar’s audio is received., with interactive elements