i am using the openai realtime api and the multilingual mode LiveKit Community #ask-ai

i am using the openai realtime api and the multili...

early-restaurant-74591

08/02/2025, 3:24 AM

i am using the openai realtime api and the multilingual model for turn detection. i want to create a textual transcription of the conversation between the user and the agent. how can i do that?

tall-airline-16756

08/02/2025, 3:24 AM

Thanks for reaching out, Jason. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

08/02/2025, 3:25 AM

To create a textual transcription of a conversation using the OpenAI real-time API and LiveKit's multilingual model for turn detection, follow these steps: 1. Install Required Plugins: Install the LiveKit turn detector and OpenAI real-time plugins using pip:

pip install "livekit-agents[turn-detector,openai]" openai

2. Initialize Components: Use Deepgram STT and OpenAI's real-time API while disabling OpenAI's built-in turn detection. 3. Configure the Agent Session: Set up the

AgentSession

with the necessary components. Here is a complete Python code example:

Copy code

from livekit.agents import (
    Agent,
    AgentSession,
    JobContext,
    WorkerOptions,
    cli,
)
from livekit.plugins import (
    deepgram,
    openai,
    silero,
)
from livekit.plugins.turn_detector.multilingual import MultilingualModel

from dotenv import load_dotenv
load_dotenv()

async def entrypoint(ctx: JobContext):
    stt = deepgram.STT(model="nova-3", language="multi")
    turn_detection = MultilingualModel()
    llm = openai.realtime.RealtimeModel(
        voice="alloy",
        turn_detection=None,
        input_audio_transcription=None,
    )
    session = AgentSession(
        turn_detection=turn_detection,
        vad=silero.VAD.load(),
        stt=stt,
        llm=llm,
    )
    agent = Agent(instructions="You are a helpful assistant.")
    await session.start(agent=agent, room=ctx.room)

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

Key Points: • Use Deepgram STT for multi-language support. • Leverage the

MultilingualModel

for turn detection. • Disable OpenAI's built-in turn detection to avoid conflicts. Ensure you have the necessary API keys for OpenAI and Deepgram set in your environment variables before running the script.

early-restaurant-74591

08/02/2025, 3:57 AM

thanks but which part of that code creates the textual transcription of the conversation?

tall-airline-16756

08/02/2025, 3:58 AM

Hey there! Just wanted to walk you through how the textual transcription of the conversation comes together with a few key parts in the code: 1. Deepgram STT: This is where the audio input gets processed, creating both interim and final transcriptions. These transcription events show up as

SpeechEvent

objects that include the text and language details. 2. OpenAI Real-time API: This API takes in audio frames via WebSocket and keeps the transcriptions updated in real-time. It handles interim transcriptions through delta updates, and once the audio finishes, it processes the final transcriptions. 3. Turn Detection: The

MultilingualModel

steps in here to spot when the conversation switches between speakers, making sure the user’s and agent’s speech are neatly separated. In the code you have, most of the transcription magic happens with the

stt

component (Deepgram STT) and the

llm

component (OpenAI real-time API). After that, the transcriptions are handed off to the

AgentSession

, which keeps the conversation flowing and outputs the transcribed text. If anything’s unclear or you want to dive deeper into any part of the transcription process, just give me a shout—I’m here to help! Sources: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/real...

early-restaurant-74591

08/02/2025, 4:01 AM

can you write me some code that, at the end of a conversation, prints the transcript, showing what the agent says, then what the user says in response, then what the agent says, etc. etc. ?

tall-airline-16756

08/02/2025, 4:01 AM

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!

2 Views

Open in Slack

Previous Next