How to transcribe audio from Livekit Rooms using Google MLKit

Thanks for reaching out, Madhav. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

How to do transcription with open source or localhost or on device functionalities

LiveKit doesn’t have built-in support for open-source or on-device transcription just yet, but don’t worry! You can definitely set this up yourself by: • Using LiveKit SDKs to capture raw audio streams from participants in real time. • Processing the audio locally with an open-source STT engine like Vosk or Whisper.cpp. • Publishing the resulting transcripts back into LiveKit using the

publish_transcription()

method. If you’d like, I’m here to help with guidance or even a code example to integrate a specific open-source STT model with LiveKit’s audio stream. Just let me know! Sources: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py | main.py | server/main.py | Transcriptions | LiveKit Docs | Groq integration guide | LiveKit Docs

okay how do i do this : • Using LiveKit SDKs to capture raw audio streams from participants in real time. • Processing the audio locally with an open-source STT engine like Vosk or Whisper.cpp. • Publishing the resulting transcripts back into LiveKit using the

publish_transcription()

method

To get started with transcription using open-source or on-device options, here’s a simple guide to help you along the way: • Capture Raw Audio Streams: You can use LiveKit SDKs to grab audio frames through the

AudioStream

class. For instance, by listening to the

track_subscribed

event, you’ll be able to process audio frames as they come in, in real-time. • Process Audio Locally: Consider using an open-source STT engine like Whisper.cpp or Vosk. With Whisper.cpp, you’ll want to preprocess the audio into mel spectrograms before running inference. If you choose Vosk, you can send raw PCM audio frames straight to the recognizer for transcription. • Publish Transcription: Once you have your transcription results, the

publish_transcription()

method lets you share them back to the LiveKit room, so everyone can see the transcriptions live. If you’d like, I’d be happy to provide a detailed code example to help you integrate Whisper.cpp or Vosk with LiveKit. Just let me know which STT engine you prefer, and I’ll guide you through the next steps! Sources: examples/other/transcription/README.md | Transcriptions | LiveKit Docs | server/main.py | plugins/openai/src/stt.ts | tools/mtmd/mtmd.cpp

do I have the AudioStream class in the Flutter SDK?

The LiveKit Flutter SDK might not have an AudioStream class like the Python SDK for capturing raw audio. Could you please let me know if you're trying to do this specifically in a Flutter application? If that's the case, I'd be happy to help explore some alternative ways to access audio streams in Flutter with you.

Yes I am doing this in a Flutter Appliation

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!