How can I make the agent only listen and transcribe audio wi LiveKit Community #ask-ai

Join Slack

How can I make the agent only listen and transcrib...

# ask-ai

colossal-airport-32984

08/04/2025, 2:42 PM

How can I make the agent only listen and transcribe audio, without responding (when using openai realtime API)?

tall-airline-16756

08/04/2025, 2:42 PM

Thanks for reaching out, Benjamin. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

tall-airline-16756

08/04/2025, 2:42 PM

To configure an agent to only listen and transcribe audio without responding when using the OpenAI Realtime API, follow these steps: 1. Initialize an

Agent

with minimal interaction capabilities:

Copy code

transcriber = Transcriber()
session = AgentSession(
    stt=openai.STT(),
    # No LLM or TTS is used since we want only transcription
)

1. Configure the session to disable audio output and enable transcription:

Copy code

await session.start(
    agent=transcriber,
    room=ctx.room,
    room_output_options=RoomOutputOptions(
        transcription_enabled=True,
        audio_enabled=False,
    ),
)

1. Set up the transcriber to handle only speech-to-text:

Copy code

class Transcriber(Agent):
    def __init__(self):
        super().__init__(
            instructions="not-needed",
            stt=openai.STT(),
        )

    async def on_user_turn_completed(self, chat_ctx: llm.ChatContext, new_message: llm.ChatMessage):
        # Simply log the transcription without generating a response
        user_transcript = new_message.text_content
        <http://logger.info|logger.info>(f"User said: {user_transcript}")
        raise StopResponse()

This configuration ensures that the agent: • Listens to audio input • Transcribes the audio to text • Publishes the transcription to the room • Does not generate any audio or text responses Key points: • No LLM (Language Model) is used in the session. • No TTS (Text-to-Speech) plugin is configured. • Audio output is explicitly disabled in

RoomOutputOptions

. • The

on_user_turn_completed

method is overridden to prevent generating responses. The transcribed text will be available through the

lk.transcription

text stream topic. Sources: Text and transcriptions | LiveKit Docs

colossal-airport-32984

08/04/2025, 2:44 PM

How about if I use the llm as openai realtime API e.g.

Copy code

llm=openai.realtime.RealtimeModel(
                model="gpt-4o-realtime-preview-2025-06-03",
                voice="sage",
                api_key=CONFIG.openai_api_key,
                turn_detection=TurnDetection(
                    type="semantic_vad",
                    eagerness="low",
                    create_response=True,
                    interrupt_response=True,
                ),
                temperature=0.6,
                input_audio_noise_reduction=InputAudioNoiseReduction(
                    type=params.audio_device_noise_reduction.value
                ),
                input_audio_transcription=InputAudioTranscription(language=lang, model="whisper-1"),
                max_session_duration=20 * 60,  # timeout to reconnect to OAI
            )

how can I make it so the agent simply listens and transcribes but does not respond?

tall-airline-16756

08/04/2025, 2:45 PM

If you want to set up the agent using the OpenAI Realtime API (

RealtimeModel

) so it just listens and transcribes without replying, here’s how you can tweak the settings: 1. Switch the modalities to text-only to turn off audio responses. 2. Turn on input audio transcription so it can convert what the user says into text. 3. Adjust the turn detection settings to stop it from generating any responses. Here’s a sample configuration to get you started:

Copy code

from livekit.plugins.openai import realtime

# Initialize the RealtimeModel with text-only modality
model = realtime.RealtimeModel(
    modalities=["text"],  # Set to text-only to disable audio responses
    input_audio_transcription={
        'model': 'whisper-1'  # Enable transcription
    },
    turn_detection={
        'type': 'server_vad',
        'threshold': 0.5,
        'prefix_padding_ms': 300,
        'silence_duration_ms': 200,
        'create_response': False,  # Set to False to prevent response generation
        'interrupt_response': False
    }
)

# Create an AgentSession with the configured model
session = AgentSession(llm=model)

Here are the key things to keep in mind: •

modalities=["text"]

makes sure the model only works in text mode, so no audio replies. •

input_audio_transcription

is turned on to transcribe what’s being said. •

turn_detection

is set up to stop the model from answering after it transcribes. This way, your agent will just listen and transcribe audio without jumping in with any responses. If you have any questions or want to tweak it further, feel free to ask!

Open in Slack

Previous Next