Hi guys im facing an issue where the agent text stream (transcription) is not synchronized with the LiveAvatar’s video and voice during streaming. The transcript appears out of alignment with what the avatar is speaking on screen (transcription appears before avatar voice + video), instead of being displayed in real time as the video is streamed. Other avatars (e.g., Tavus and Anam) correctly support synchronized text, video, and audio playback. LiveAvatar should exhibit the same behavior, with the agent transcription aligned to the avatar’s spoken words as they occur. Can anyone assist with this. Furthermore I am unable to detect the agent events as the websocket is wrapped in the plugin. can anyone guide about this @refined-appointment-81829 @lively-winter-88668

I'm not sure what TTS you have, but did you try https://docs.livekit.io/agents/multimodality/text/#tts-aligned-transcriptions ? Otherwise, transcription will be applied at the sentence-level.

@lively-winter-88668 so the audio is coming with the avatar video (liveavatar) would it be synced with text in that case? also we are using gpt-4o mini tts

I see what you mean now, the transcription is misaligned with LiveAvatar isn't it. I see LiveAvatar uses https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-liveavatar/livekit/plugins/liveavatar/avatar.py#L134 whereas others such as Anam use https://github.com/livekit/agents/blob/main/livekit-plugins/livekit-plugins-anam/livekit/plugins/anam/avatar.py#L116, not sure if that is significant. I see you also opened https://github.com/livekit/agents/issues/4406 so I'll keep an eye on that.

@gray-teacher-71553 any thoughts?

@gray-teacher-71553 I also can't see a method to receive events emitted by the liveavatar, any advice on that? Im also attaching the synchronization bug here for reference: https://github.com/livekit/agents/issues/4406

Hi Rida, I'll respond to the synchronization issue on GitHub, but I do think results may vary for each person as LiveAvatar is more synchronized for me than Tavus. For the emitted events, what is your use case and which events are you looking for? The equivalent to LiveAvatar's

agent.speak_started

would be the

AgentState

changing to

"speaking"

in our

agent_state_changed

event, and

agent.speak_ended

would be the

AgentState

changing to

"listening"

@gray-teacher-71553 so I had checked and agent state turns to "speaking" about 1-2 seconds before the avatar starts to speak. so the speaking event aligns with the transcription however the avatar speaks noticeably late