Real-time translation agent using LiveKit Agent without turn detection
Hello LiveKit team
I’m currently testing LiveKit Agent in Python and would like to know if it’s possible (or planned) to support real-time translation agents via streaming , without relying on turn detection.
My goal is to build an agent that listens continuously to the caller and repeats everything the user says in another language , in real time — similar to a simultaneous interpreter.
So far, using AgentSession, it seems that the pipeline is turn-based, and the audio is processed only after a speech segment is detected and closed by the VAD (Voice Activity Detection). This causes translation delays and prevents true streaming behavior.
Ideally, I’d like to create a continuous stream pipeline combining:
STT (e.g., gpt-4o-mini-transcribe) for real-time transcription,
LLM for translation or language conversion,
TTS (e.g., gpt-4o-mini-tts) for real-time voice output —
all in a streaming loop , without waiting for turn completion.
Is there any way to achieve this with the current AgentSession or RealtimeModel APIs, or should this be implemented directly via the lower-level Realtime API / AudioStream interfaces?
Any guidance or roadmap information for this kind of continuous translation agent would be greatly appreciated