Hey, is there some existing example on what is the...
# agents
m
Hey, is there some existing example on what is the best way to implement avatar in livekit? I've saw this example: https://github.com/livekit/agents/tree/main/examples/avatar_agents/audio_wave which generate a simple frame from the audio. All other examples there are simple 3rd party and doesn't actually explain how to self implement it. I've tried doing something myself based on wave example, but it is very laggy and doesn't work. I find this example a bit weird as it require the avatar rendering be almost realtime, while most existing approaches are not. Has anyone tried doing something like this and manage making it look smooth? Are there any examples? Is there a different way to do this?
l
We support a number of avatar providers out of the box, as listed at https://docs.livekit.io/agents/models/avatar/. The source code for how these plugins are integrated with LiveKit is available in our agents repository, for example Tavus is at https://github.com/livekit/agents/tree/main/livekit-plugins/livekit-plugins-tavus.
m
@lively-winter-88668 Thank you for your response. I know there are the avatar plugins, however, I would like to create something of my own for several reasons. I already have my code for generating video given the audio stream, and I'm trying to understand how can I integrate my code with livekit, as simply replacing the wave generator doesn't seem the way I'm intended doing it. Is there some guide or an example on how this can be done? Or as far as it concern livekit, I'm forced to pay to one of the 3rd parties if I want avatar, and prevented from implementing something of my own?
l
Hey James, I thought you were essentially asking how to implement your avatar as a plugin in LiveKit, so my suggestion was to copy one of the existing avatar deployments, that's why I gave the link to Tavus. There isn't a specific guide for this unfortunately but all the code is open source, there is no requirement to pay for either LiveKit, or 3rd parties. Please let me know if I have misunderstood.
m
Thanks. I don't mind if I use it as plugin or not. The issue in the tavus example you sent, that its launching a worker and simply send an api request to Tavus with the room details. What I'm missing, is how the part beyond this api call look like. I assumed that on Tavus servers there is something which reminds the wave example I sent in the first message. In the example we have the dispatcher that get the room details, connect to it, and connect the actual agent worker to the room, allowing the data streaming. However, in the avatar_runner code, it basically get audio chunks (generated as a response) and return the frame for this audio chunk. In order for it to look smooth, the runtime of the generated frame should be extremely fast, not to create a delay in the stream. However, most avatar models cannot generate a frame in ~10ms. In addition, when I trying the #C088ZNU7QQ5 channel, it wrote as a response that this is not the way to generate avatar (inside of the agent process). However, didn't say what is the correct way. For those reason, I tried to consult with you, to understand what I'm doing wrong. That is, what is the correct way to add agent to the room, which doesn't have realtime runtime? Thanks again for all of your help.
l
m
Thanks anyway. I will try to investigate what is the correct way to implement the agent itself.
Hey @refined-appointment-81829, tagging you in the thread as you asked. Maybe you will have an idea. Thanks!
r
Hi @most-match-32106 What you are seeing is how our Avatar integration works. @quaint-ambulance-32873 may have some good advise on how to implement a custom plugin. https://github.com/livekit/agents/tree/main/examples/avatar_agents/audio_wave is a good starting point and is just a very simple example to show how to publish to the room. avatar_runner.py is essentially what your service will need to do to publish the audio/video to the room I am not sure why you say > I find this example a bit weird as it require the avatar rendering be almost realtime, while most existing approaches are not. The avatar service by its very nature needs to be realtime. To mask the lag the audio from the avatar service is what the room hears instead of the audio directly from the agent. This helps to mask the delay that is required for rendering the avatar video steram. Here are the basics of what you will need to do: Overview Avatar services in LiveKit follow a plugin architecture where each service provides an
AvatarSession
class that manages the connection between the voice agent and the avatar service. The avatar appears as a separate participant in the LiveKit room, receiving audio from the agent and publishing video back. Implementation Steps 1. Create Plugin Structure Create a new plugin package with the following structure:
Copy code
# livekit/plugins/yourservice/__init__.py  
from .avatar import AvatarSession  
from .api import YourServiceException  
  
__all__ = ["AvatarSession", "YourServiceException"]
2. Implement AvatarSession Class The
AvatarSession
class is the core component that manages the avatar lifecycle:
Copy code
class AvatarSession:  
    """Your Service avatar session"""  
      
    def __init__(  
        self,  
        *,  
        api_key: NotGivenOr[str] = NOT_GIVEN,  
        api_url: NotGivenOr[str] = NOT_GIVEN,  
        avatar_participant_identity: NotGivenOr[str] = NOT_GIVEN,  
        avatar_participant_name: NotGivenOr[str] = NOT_GIVEN,  
        # Add your service-specific config here  
    ) -> None:  
        # Initialize your API client and configuration
3. Implement the start() Method The
start()
method handles the connection flow:
Copy code
async def start(  
    self,  
    agent_session: AgentSession,  
    room: rtc.Room,  
    *,  
    livekit_url: NotGivenOr[str] = NOT_GIVEN,  
    livekit_api_key: NotGivenOr[str] = NOT_GIVEN,  
    livekit_api_secret: NotGivenOr[str] = NOT_GIVEN,  
) -> None:  
    # 1. Get local participant identity  
    job_ctx = get_job_context()  
    local_participant_identity = job_ctx.local_participant_identity  
      
    # 2. Create LiveKit token for avatar participant  
    livekit_token = (  
        api.AccessToken(api_key=livekit_api_key, api_secret=livekit_api_secret)  
        .with_kind("agent")  
        .with_identity(self._avatar_participant_identity)  
        .with_name(self._avatar_participant_name)  
        .with_grants(api.VideoGrants(room_join=True, room=room.name))  
        .with_attributes({ATTRIBUTE_PUBLISH_ON_BEHALF: local_participant_identity})  
        .to_jwt()  
    )  
      
    # 3. Call your service API to start session  
    session_data = await self._api.start_session(  
        livekit_url=livekit_url,  
        livekit_token=livekit_token,  
        # Your service-specific parameters  
    )  
      
    # 4. Configure audio output routing  
    agent_session.output.audio = DataStreamAudioOutput(  
        room=room,  
        destination_identity=self._avatar_participant_identity,  
        sample_rate=SAMPLE_RATE,  
        wait_remote_track=rtc.TrackKind.KIND_VIDEO,  
    )
4. Create API Client Implement an API client to communicate with your avatar service:
Copy code
class YourServiceAPI:  
    async def start_session(  
        self,  
        livekit_url: str,  
        livekit_token: str,  
        # Your parameters  
    ) -> dict:  
        # Make HTTP request to your service  
        async with self._http_session.post(  
            f"{self._api_url}/v1/session",  
            json={  
                "livekit_url": livekit_url,  
                "livekit_token": livekit_token,  
                # Your service-specific config  
            }  
        ) as response:  
            return await response.json()
Key Implementation Details Token Generation All avatar services use the
ATTRIBUTE_PUBLISH_ON_BEHALF
attribute to allow the avatar participant to publish tracks on behalf of the agent avatar.py:88-99 . Audio Output Routing Two main patterns exist for audio routing: • `DataStreamAudioOutput`: Streams audio directly to the avatar participant avatar.py:116-121 • `QueueAudioOutput`: Buffers audio for synchronized playback avatar.py:127-131 Sample Rate Configuration Different services use different sample rates: • LiveAvatar: 24000 Hz avatar.py:30 • Anam: 24000 Hz avatar.py:25 • AvatarTalk: 16000 Hz avatar.py:16 TypeScript Implementation For TypeScript/JavaScript implementations, follow the same pattern Best Practices 1. Error Handling: Implement proper error handling with custom exceptions __init__.py:6-7 2. Logging: Add structured logging for debugging avatar.py 3. Retry Logic: Use
APIConnectOptions
for configurable retry behavior avatar.py:112-142 4. Cleanup: Implement proper session cleanup on shutdown avatar.py:85-90 Notes • All avatar services create a separate participant in the LiveKit room • The
publish_on_behalf
attribute is crucial for allowing the avatar to publish tracks • Audio routing is handled through specialized audio output classes • Each service may have unique configuration parameters and API endpoints • Consider both Python and TypeScript implementations for broader compatibility
✅ 1
q
what s the frame per seconds your avatar is doing atm?
m
@refined-appointment-81829 thanks for the response! @quaint-ambulance-32873 My avatar generation runs at about 21~22 fps
q
the bit that you need to implmene at your server is this bit https://github.com/livekit/agents/blob/main/examples/avatar_agents/audio_wave/avatar_runner.py you need to implement aiter for generating video frame and audio frame, and also send when audio segement end, AudioSementEnd then you need to implement push_audio, this is where the audio comes from livekit and you feed these audio into the video generation pipeline
that should get you 90% there, but there is also clear_buffer when it is triggered when there is interruption so you can clear the buffer in your video generation pipeline
m
Thanks for the response, this is indeed what I did. However, I get something that "stutter" and not very smooth. Generally when I took the wave example and just add similar delay as my generation I get the same "stutter", which tells me that 21~22 fps is too slow to work properly. There is always delay between end of the person audio and until we start to send reply, so I thought maybe about option to start buffer the response, this way having less latency when sending the video. But I didn't find yet how can I implement something like this. I guess the other option is to make the generation even faster, but so far couldn't find a way to do it neither.
q
you can think about maybe adding some inbetween frames using frame interpolation which is availabe in ffmpeg to artificially increase your frame rate (the one in ffmpeg can be run in CPU)
🙏 1
m
thanks for the help