https://livekit.io logo
Join Slack
Powered by
How can I access raw audio input (for example, mic...
# ask-ai
w
How can I access raw audio input (for example, microphone audio from a participant) in livekit or livekit agent
t
Thanks for reaching out, maciej.jakub. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
You're asking how to access raw microphone audio from a participant in LiveKit or a LiveKit Agent — you can do this using the AudioStream APIs to read raw PCM audio frames from a subscribed audio track. Key points and a minimal example (Agent / server-side): • Subscribe to the participant's audio track (handle the track_subscribed event). • Wrap the subscribed track with 
rtc.AudioStream
. • Iterate the stream to receive 
AudioFrame
objects. Each frame includes 
frame.data
(PCM 16-bit signed integers), 
frame.sample_rate
, and 
frame.num_channels
. • Buffer or process frames as needed (real-time processing, STT feeding, saving short snippets, etc.). Use helper like 
combine_audio_frames
to produce WAV bytes for short captures. Example (Python Agent):
Copy code
from livekit import rtc

@room.on("track_subscribed")
async def on_track_subscribed(track, publication, participant):
    if track.kind == rtc.TrackKind.KIND_AUDIO:
        stream = rtc.AudioStream(track)
        async for frame_event in stream:
            frame = frame_event.frame
            raw_pcm = frame.data          # bytes of 16-bit PCM samples
            sr = frame.sample_rate
            channels = frame.num_channels
            # process raw_pcm...
        await stream.aclose()
Notes: • Frames are PCM 16-bit signed (little-endian). Use 
sample_rate
and 
num_channels
from the frame for resampling/processing. • For short snippets, collect frames until a time threshold, then combine to WAV (e.g., 
combine_audio_frames(...).to_wav_bytes()
). • The same AudioStream approach is available in client SDKs for local/raw track access if you need client-side processing. If you want, tell me: • Which SDK/language (Python Agent, Node, Web, or mobile) you're using, • Whether you need continuous real-time processing or just short recordings, and I’ll give a tailored code example and next steps. Sources: Processing raw media tracks | LiveKit docs | Subscribing to tracks | LiveKit docs | Job lifecycle | LiveKit docs | Capturing the First Few Seconds of a Call for Processing | Camera & microphone | LiveKit docs
w
Ok, and how to do it on the agent side? So I have an LiveKitAgent, and how to obtain this raw audio in it?
t
You're using a LiveKit Agent — to give the exact code and steps, which Agent language/SDK are you using: Python, Node, or another language? If Python, I can provide a compact example showing: 1. Handling 
track_subscribed
in the agent. 2. Wrapping the track with 
rtc.AudioStream
. 3. Iterating 
AudioFrame
objects and reading 
frame.data
, 
frame.sample_rate
, 
frame.num_channels
. 4. Optionally combining frames into a WAV for short snippets. Which language do you want the example in?
w
Python
t
Great — below is a concise Python LiveKit Agent example showing how to obtain raw microphone audio from a participant's published audio track, read PCM frames, and optionally combine a short snippet into WAV bytes.
Copy code
from livekit import rtc
from livekit.agents import combine_audio_frames

# Attach handler to the Room/Agent instance (assumes `room` is your agent Room)
@room.on("track_subscribed")
async def on_track_subscribed(track: rtc.Track, publication: rtc.RemoteTrackPublication, participant: rtc.RemoteParticipant):
    # Only handle audio tracks
    if track.kind != rtc.TrackKind.KIND_AUDIO:
        return

    # Wrap the incoming track to get an async iterator of AudioFrame events
    stream = rtc.AudioStream(track)

    buffer = []
    try:
        async for frame_event in stream:
            frame = frame_event.frame
            # Raw PCM bytes (16-bit signed little-endian)
            raw_pcm = frame.data
            sample_rate = frame.sample_rate
            channels = frame.num_channels
            timestamp_ms = frame.timestamp  # typically in milliseconds

            # Example: collect frames for first N ms (e.g., 5000ms)
            buffer.append(frame)
            total_ms = buffer[-1].timestamp - buffer[0].timestamp if len(buffer) > 1 else 0
            if total_ms >= 5_000:
                # Combine buffered frames into a single audio object and get WAV bytes
                combined = combine_audio_frames(buffer)
                wav_bytes = combined.to_wav_bytes()  # ready-to-save WAV
                # Do something with wav_bytes (STT upload, save, etc.)
                buffer.clear()

            # Or process `raw_pcm` per-frame for real-time pipelines (STT, VAD, etc.)

    finally:
        # Clean up when done
        await stream.aclose()
Key details: • 
frame.data
contains raw PCM 16-bit signed little-endian samples. • Use 
frame.sample_rate
and 
frame.num_channels
for resampling or channel handling. • 
combine_audio_frames
helps convert buffered frames into WAV bytes for short recordings. If you'd like, I can: • Provide a complete runnable agent snippet including room initialization and authentication, or • Show how to stream frames into a specific STT service (give the STT provider you use).
5 Views
Open in Slack
PreviousNext
Powered by