Hi there! To help you consume audio from a participant in a LiveKit room using Python, here’s a friendly step-by-step guide you can follow: 1. Connect to the LiveKit Room: Start by using the LiveKit Python SDK to connect to a room with an access token. Here’s a simple example:

from livekit import api

token = (
    api.AccessToken()
    .with_identity("python-bot")
    .with_grants(api.VideoGrants(room_join=True, room="my-room"))
    .to_jwt()
)

1. Set Up Event Listener for Track Subscription: You’ll want to listen for the

track_subscribed

event to know when a remote participant’s audio track is available. 2. Create an AudioStream to Consume Audio Frames: Here’s how you can do it:

@room.on("track_subscribed")
def on_track_subscribed(track, publication, participant):
    if track.kind == rtc.TrackKind.KIND_AUDIO:
        audio_stream = rtc.AudioStream(track)
        asyncio.create_task(handle_audio_stream(audio_stream))

async def handle_audio_stream(audio_stream):
    async for audio_frame_event in audio_stream:
        frame = audio_frame_event.frame
        audio_data = np.frombuffer(frame.data, dtype=np.int16)
        # You can process audio_data here as needed
    await audio_stream.aclose()

1. Optional: Enable Auto-Subscribe: If you want to make things even smoother, set

auto_subscribe=True

when connecting, so audio tracks subscribe automatically. 2. Complete Example: Here’s a minimal, friendly example that consumes remote audio and plays it using `sounddevice`:

import asyncio
import sounddevice as sd
import numpy as np
from livekit import rtc, api

SAMPLE_RATE = 48000
CHANNELS = 1

async def play_audio(audio_stream):
    def audio_callback(outdata, frames, time, status):
        if not audio_queue.empty():
            data = audio_queue.get_nowait()
            outdata[:len(data)] = data.reshape(-1, 1)
            outdata[len(data):] = 0

    audio_queue = asyncio.Queue()

    with sd.OutputStream(samplerate=SAMPLE_RATE, channels=CHANNELS, dtype='int16', callback=audio_callback):
        async for event in audio_stream:
            frame_data = np.frombuffer(event.frame.data, dtype=np.int16)
            await audio_queue.put(frame_data)
        await audio_stream.aclose()

async def main():
    room = rtc.Room()
    token = api.AccessToken().with_identity("bot").with_grants(api.VideoGrants(room_join=True, room="my-room")).to_jwt()

    @room.on("track_subscribed")
    def on_track_subscribed(track, _, participant):
        if track.kind == rtc.TrackKind.KIND_AUDIO:
            asyncio.create_task(play_audio(rtc.AudioStream(track)))

    await room.connect("<wss://your-room.livekit.cloud>", token)
    await asyncio.sleep(3600)

asyncio.run(main())

I hope this helps you get started with consuming audio in your LiveKit room! If you have any questions or want me to walk you through any part in more detail, just let me know — I’m here to help! Sources: Job lifecycle | LiveKit Docs | README.md | terminal-client/client.py | stream_audio.py

Let’s work together to figure out which part of your audio processing flow (VAD, STT, LLM, TTS) might be taking the most time. LiveKit offers helpful metrics for each component that can guide us. Here’s a friendly walkthrough on how you can measure and optimize performance: • Measuring Performance: • VAD Metrics: You can use the

metrics_collected

event to see idle time, inference duration, and how many inferences are happening. • STT Metrics: Keep an eye on audio duration, request duration, and whether streaming is enabled. • LLM Metrics: Check the time to first token (TTFT), total response time, and token usage to understand efficiency. • TTS Metrics: Look at time to first byte (TTFB), synthesis duration, and audio duration for insights. Here’s an example to capture these metrics:

@agent.on("metrics_collected")
def on_metrics(metrics):
    print(metrics)

• Optimizing Performance: • Reduce Latency: Try deploying agents closer to your LLM, STT, and TTS services to cut down network delays. • Enable Streaming: Streaming for STT and TTS can help you process data as it comes in, speeding things up. • Adjust Endpointing: Tweak

minEndpointingDelay

and

maxEndpointingDelay

settings to fine-tune response timing. • Use Efficient Models: Choosing lightweight models can make your processing faster and smoother. If you want to dive deeper, the LiveKit Metrics Documentation is a great resource. And of course, I’m here to help if you’d like a hand implementing any of these steps! Sources: agents/src/voice/agent.ts | livekit-agents/livekit/agents/voice/audio_recognition.py | Logs, metrics, and telemetry | LiveKit Docs | Speech & Audio Processing | livekit/livekit_composite | DeepWiki | metrics/send-metrics-to-3p/send_metrics_to_3p.py