My use case is to build a system where two `SIP pa...
# agents
n
My use case is to build a system where two
SIP participants
join the same room, each speaking in their own language, and each hearing in their own language. To achieve this, after both participants join, I unsubscribe each participant’s microphone track from the other (using the API since there is no control over remote participant track subscriptions). Then I publish two additional audio tracks that contain the translated audio. The local participant (an agent) subscribes to both microphone audio tracks, performs STT on the incoming audio, translates the text into the other participant’s language using an LLM, converts it to speech using TTS, and finally publishes the translated audio. I want to know:
Is there a better or more efficient way to implement this?
I also considered handling this with two agent sessions inside the same room, but the second agent session does not work correctly and causes issues.
@lively-winter-88668 @refined-appointment-81829 Could you please help me finalize the approach? I want to understand whether my design is correct, and also what changes are needed if I want to handle this using two agent sessions inside the same room. Currently, the second agent session doesn’t work properly and causes issues. I’m also unsure whether I need to use VAD (Voice Activity Detection) to segment the incoming audio before sending it to STT, or if it’s optional in this scenario. I referred to the following example:
<https://github.com/livekit/agents/blob/main/examples/other/translation/multi-user-translator.py>
My flow is similar — I already have STT → LLM → TTS implemented like in the example. I just need guidance on the recommended architecture, and if VAD is required, please point me to any example or reference I can follow.
l
Yes, you should use a VAD and the multilingual turn detection model. I recommend only having a single AgentSession (your first approach), though I note you have also raised https://livekit-users.slack.com/archives/C07FRP66XGE/p1765785797713609 so not sure if you had issues with that approach. Did you also see https://github.com/livekit-examples/live-translated-captioning ? It's a little old, but does seem very close to what you are looking to achieve.
n
but can you pls help me with this issue https://livekit-users.slack.com/archives/C07FRP66XGE/p1765785797713609 what should i do ?
@lively-winter-88668