Using the Flutter Livekit SDK, can I get the PCM F...
# ask-ai
g
Using the Flutter Livekit SDK, can I get the PCM Frames from the Livekit room to feed into other services
t
Thanks for reaching out, Madhav. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
I'm glad you're exploring how to extract PCM frames from a LiveKit room using the Flutter SDK! There are a few ways to approach this, depending on what you're aiming to achieve: • Using AudioVisualizer: The Flutter SDK comes with an AudioVisualizer feature that works with raw PCM data internally to create audio levels or FFT bands. While it doesn't give direct access to raw PCM frames, it does show that the SDK can handle raw audio data. • Server-Side TrackEgress: If you need full PCM streams, LiveKit’s TrackEgress lets you stream audio tracks to a WebSocket server in raw PCM format. This is a great option for real-time tasks like processing or transcription. • Custom Flutter Plugin: Another path is to customize the Flutter SDK by extending its native code to expose raw audio frames. This involves capturing audio at the platform level and sending it to Dart through platform channels. I'd love to hear more about your specific use case—whether it's transcription, voice activity detection, recording, or something else—so I can help guide you to the best solution. Feel free to share more details! Sources: Universal Egress | shared_cpp/fft_processor.cpp | Track egress | LiveKit Docs | Universal Egress | livekit-agents/livekit/agents/utils/audio.py
g
I want to do transcription and then show it to the user
t
For transcription, a great way to get started is by using LiveKit's TrackEgress feature to stream audio tracks to a WebSocket server in raw PCM format. This method lets you process the audio in real-time with transcription services like Google Speech-to-Text or AWS Transcribe, making things smoother for you. Here’s a simple way to set it up: 1. First, configure TrackEgress to send the audio track from your LiveKit room to a WebSocket server. The audio will be streamed in
pcm_s16le
format. 2. Next, set up a WebSocket server to receive this audio data and forward it to your chosen transcription service. 3. Finally, you can display the transcribed text to users right in your Flutter app. If you’d like, I’m here to help with detailed steps on setting up TrackEgress or integrating the transcription service—just let me know!
g
Is it possible to handle this on device to avoid using a server
t
It’s definitely possible to handle transcription directly on your device! To do this, you’d need to integrate an on-device transcription library or SDK. Here are a few options you might find helpful: • Google Speech-to-Text API (On-Device): Google’s on-device ML Kit for speech recognition is a great choice. It works offline, so you won’t have to worry about server dependencies. • Open-Source Libraries: Libraries like Vosk or Picovoice can also handle on-device transcription and work well with Flutter through platform channels. • Custom Flutter Plugin: Another approach is to modify the Flutter LiveKit SDK to expose raw PCM frames, then use those frames with an on-device transcription library for processing. If you’d like, I’d be happy to guide you through integrating one of these libraries or help you modify the Flutter SDK to access raw PCM frames for transcription. Just let me know how you’d like to proceed!
g
how to transcribe on device, using, for example, sherpa onnx
t
I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!
g
How to transcribe using Google's MLKIt