we have integrated livekit, when we show customer demo on google meet, the audio track merges and the agent doesn't pick the voice at all

Thanks for reaching out, Arjun. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

For the text that appears below as one talks, the transcript generated for what I said seemed very wrong a lot of the time. Yet, the agent seemed to have heard what I said accurately and gave competent answers. If you looked at the text below as it went though, the thing looked pretty embarrassing. Is there a fix coming for that?

Improving transcription accuracy in LiveKit is definitely achievable by fine-tuning the configuration of your Speech-to-Text (STT) provider. Let me walk you through some helpful steps: • Choose an accurate STT provider: Providers like Google Cloud STT and Deepgram are great choices known for delivering high accuracy in real-time transcription. • Enable enhanced accuracy settings: For instance, Speechmatics offers an

operating_point

setting that lets you prioritize accuracy over latency, which can make a real difference. • Use custom vocabulary: Adding domain-specific terms with the

additional_vocab

parameter can really boost recognition of those unique or uncommon words. • Optimize audio quality: Clear audio is key! Using noise cancellation or tweaking noise reduction settings in your STT plugin can help a lot. • Adjust audio chunking: Finding the right frame size and encoding (like 20ms frames) helps balance latency and accuracy smoothly. If you’re working with a particular STT provider, I’d be more than happy to guide you through configuring these settings step-by-step. Just let me know if you'd like some extra help with implementation or testing—I’m here to support you! Sources: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py | Speechmatics integration guide | LiveKit Docs | Live Conversations with AI using ChatGPT and WebRTC | Live Conversations with AI using ChatGPT and WebRTC | README.md

we are using realtime models

Since you're working with real-time models for transcription, I’d love to share some friendly tips to help improve accuracy: • Provider Selection: It’s great to choose a provider that’s really tuned for real-time transcription, like Google Cloud STT or Deepgram—they’re known for being both fast and accurate. • Enhanced Accuracy Settings: If you’re using Speechmatics, try setting the

operating_point

to

ENHANCED

. It might add a little latency, but it usually boosts accuracy nicely. • Custom Vocabulary: Adding domain-specific terms through the

additional_vocab

parameter can really help the system recognize specialized words or phrases better. • Audio Quality: Using noise cancellation and ensuring your audio input is clear can make a big difference. If your STT provider supports noise reduction settings, configuring those can reduce errors too. • Audio Chunking: Choosing the right frame sizes—like around 20ms—helps balance latency and accuracy during real-time streaming. If you’d like, I’m here to help you set these up for your specific provider or walk you through testing the changes. Just let me know!

yes, share the code config

I'm happy to share an example configuration that can help improve your transcription accuracy using Speechmatics and Google Cloud STT. Let's walk through it together: Speechmatics Configuration

const config = {
  operating_point: 'ENHANCED', // Prioritize accuracy over latency
  additional_vocab: ['custom', 'terms', 'specific', 'to', 'your', 'domain'], // Add domain-specific vocabulary
  max_delay: 500 // Adjust delay for better accuracy
};

You’ll want to pass this configuration when you initialize the Speechmatics transcription service. Google Cloud STT Configuration

const config = {
  encoding: 'LINEAR16',
  sampleRateHertz: 16000,
  languageCode: 'en-US',
  enableAutomaticPunctuation: true, // Improves readability
  model: 'default', // Use 'video' or 'phone_call' for specific use cases
  useEnhanced: true // Enable enhanced models for better accuracy
};

This configuration should be passed when setting up your Google Cloud STT client. If you’re working with a different provider or would like some help integrating these settings into your application, please don’t hesitate to reach out—I’m here to help!