Thanks — you’re asking which STT providers in the LiveKit ecosystem support speaker diarization. Good question. Two providers support diarization: • Speechmatics — Supports speaker diarization. You can enable it with

enable_diarization=True

and format the transcription output to include speaker IDs. Documentation: https://docs.livekit.io/agents/models/stt/plugins/speechmatics • AssemblyAI — Offers diarization/turn-detection features via its custom endpointing model. You can enable turn detection by setting

turn_detection="stt"

in the

AgentSession

constructor and format output for speaker identification. Documentation: https://docs.livekit.io/agents/models/stt/plugins/assemblyai If you want, I can help with next steps: • Show example configuration for either provider (Speechmatics or AssemblyAI). • Explain differences in output format (how speaker IDs appear). • Recommend which option might fit your use case based on expected number of speakers, language, or latency requirements. Which provider would you like an example for? Sources: Speechmatics STT plugin guide | LiveKit Documentation | AssemblyAI plugin guide | LiveKit Documentation

Thanks — you’re asking whether Deepgram and Soniox support speaker diarization. Yes — both providers support diarization in the LiveKit ecosystem. • Deepgram — Diarization is available through Deepgram’s STT API. In the LiveKit Deepgram plugin you enable it by setting

diarize: true

. When enabled, transcription segments are labeled with speaker identifiers (e.g., “Speaker A”, “Speaker B”) so you can separate who spoke when. Documentation: https://livekit.io/field-guides/agents • Soniox — Soniox also supports speaker diarization. In the LiveKit Soniox plugin you enable diarization via the

enable_speaker_diarization

option in

SonioxTranscribeStream

. Soniox returns system-assigned speaker labels (e.g., “S1”, “S2”) for different speakers in the audio stream. Documentation: https://docs.livekit.io/agents/models/stt/plugins/soniox If you’d like, I can: • Provide example configuration snippets for either Deepgram or Soniox. • Show sample transcription output with speaker labels so you can see the format. • Recommend which provider may fit better based on expected speaker count, languages, or latency needs. Which of those would help you next? Sources: Agents - Field Guides | LiveKit | Soniox STT plugin guide | LiveKit Documentation

Thanks — I’ll show a concise example of how to enable speaker diarization with Soniox in the LiveKit ecosystem and what to expect from the output. Example: enabling diarization with Soniox in a LiveKit transcribe stream 1. Configuration (JSON-style options you pass to the Soniox transcribe stream):

{
  "provider": "soniox",
  "options": {
    "enable_speaker_diarization": true,
    "language_code": "en-US",
    "sample_rate_hz": 48000,
    "encoding": "pcm16"
  }
}

1. Example usage sketch (pseudo-code / JS-like flow)

// Create / configure Soniox transcribe stream with diarization enabled
const sonioxOptions = {
  enable_speaker_diarization: true,
  language_code: "en-US",
  sample_rate_hz: 48000,
  encoding: "pcm16"
};

const transcribeStream = createSonioxTranscribeStream(sonioxOptions);

// Attach audio source (LiveKit participant audio, file, or stream)
participantAudio.pipe(transcribeStream);

// Listen for transcription events
transcribeStream.on("transcript", (result) => {
  // result contains segments with speaker labels
  console.log(JSON.stringify(result, null, 2));
});

1. Typical transcription output structure (what to expect) • The transcription result includes time-aligned segments. • Each segment contains a speaker label assigned by Soniox (e.g.,

S1

,

S2

), start/end timestamps, and the transcribed text. Example output (simplified):

{
  "transcript": "Hello, thanks for joining. Hi — great to be here.",
  "segments": [
    {
      "speaker": "S1",
      "start_time": 0,
      "end_time": 2.4,
      "text": "Hello, thanks for joining."
    },
    {
      "speaker": "S2",
      "start_time": 2.5,
      "end_time": 4,
      "text": "Hi — great to be here."
    }
  ]
}

Next steps I can help with • Provide a full, ready-to-run code sample in your preferred language (Node.js, Python, or server-side example). • Show how to map Soniox speaker labels (

S1

,

S2

) to known participant identities in LiveKit (e.g., by using timestamps and participant audio tracks). • Tune diarization settings or handle edge cases like overlapping speech. Which of those would you like me to provide next?

Thanks — I’ll show a focused, ready-to-run Python example that configures Soniox STT with speaker diarization in a LiveKit Agent. Prerequisites • Install the Soniox-enabled agents package:

uv add "livekit-agents[soniox]~=1.3"

• Set your Soniox API key in a .env file:

SONIOX_API_KEY=your_soniox_api_key

Example: Soniox STT with diarization in a LiveKit Python agent

from dotenv import load_dotenv
from livekit.agents import AgentSession, JobContext, AgentServer, cli
from livekit.plugins import soniox
import asyncio

load_dotenv()

server = AgentServer()

@server.rtc_session(agent_name="diarization-agent")
async def session_entrypoint(ctx: JobContext):
    # Configure Soniox STT with diarization enabled
    stt = soniox.STT(
        params=soniox.STTOptions(
            enable_diarization=True,   # <-- enable speaker diarization
            model="stt-rt-v3",         # optional: choose model
        )
    )

    # Create an AgentSession with the STT provider
    session = AgentSession(
        stt=stt,
        # add llm/tts/etc. here if required
    )

    # Listen for transcriptions
    @session.on("user_input_transcribed")
    def on_transcript(transcript):
        if transcript.is_final:
            # speaker_id is populated when diarization is enabled
            speaker = transcript.speaker_id or "unknown"
            print(f"[Speaker {speaker}] {transcript.text}")

    # Start the session and connect to the LiveKit room
    await session.start(
        room=ctx.room,
        agent=None
    )

    await ctx.connect()

if __name__ == "__main__":
    cli.run_app(server)

What to expect • Each final transcription event includes a

speaker_id

field (e.g., "S1", "S2") when diarization is enabled. • Use timestamps on the transcript segments if you need to align speaker labels to audio tracks or map speakers to known participants. If you’d like next: • I can provide a sample that maps Soniox

speaker_id

values to LiveKit participant identities using audio track timestamps. • Or produce a complete runnable example showing a participant audio stream piped into the transcribe stream. Which would help you most? Sources: Soniox STT plugin guide | LiveKit Documentation

Thanks — you’re asking whether Soniox supports the more advanced diarization flags you listed (for example

detect_primary_speaker

,

suppress_background_speaker

, and separate

primary_format

/

background_format

) and what alternatives exist. Short answer • Soniox’s LiveKit STT plugin does not support those specific options. You cannot enable

detect_primary_speaker

or

suppress_background_speaker

, nor configure separate

primary_format

and

background_format

parameters for Soniox in LiveKit. What Soniox does support • Soniox supports basic speaker diarization (identifying different speakers) and returns speaker labels (e.g.,

S1

,

S2

) when diarization is enabled. • You can apply a single-format template for active speaker output (for example via a

speaker_active_format

-style parameter), but there is no built-in way to distinguish formatting or behavior between a “primary” speaker and background speakers. Why this matters compared to Speechmatics • Speechmatics offers richer diarization-related options in LiveKit (things like diarization toggles and more flexible output formatting). That makes Speechmatics a better fit if you need built-in features to prefer or suppress a primary speaker or separate primary/background formatting. Practical alternatives • Use Speechmatics if you need built-in controls for primary-speaker preference and distinct formatting behavior. • Implement post-processing in your agent to emulate those features with Soniox: ◦ Keep counts/timestamps per

speaker_id

to detect the most active (primary) speaker over a time window. ◦ Suppress transcripts from low-activity speakers (treat them as background) before presenting or storing results. ◦ Apply different output templates in your code for “primary” vs “background” speakers (for example wrap primary speech in one tag and background speech in another). • If you want, I can provide a sample Python snippet that: ◦ Maps Soniox `speaker_id`s to a running activity score, ◦ Flags the primary speaker over a sliding window, ◦ Emits formatted output using different templates for primary vs background speakers. Which would help you most — a Speechmatics config example, or a Python post-processing sample for Soniox to emulate those options? Sources: Soniox STT plugin guide | LiveKit Documentation | Speechmatics STT plugin guide | LiveKit Documentation

Thanks — I’ll give you a complete, focused Python example that sets up a LiveKit Agent using Soniox for STT with speaker diarization and Groq as the LLM. This example shows how to enable diarization, capture speaker IDs on final transcripts, and send those transcripts to the Groq LLM while preserving speaker identifiers. Prerequisites • Python 3.10+ • Install required packages:

pip install "livekit-agents[soniox,groq,silero,cartesia]" python-dotenv

• Create a .env file with these keys:

SONIOX_API_KEY=your_soniox_api_key
  GROQ_API_KEY=your_groq_api_key
  LIVEKIT_URL=your_livekit_url
  LIVEKIT_API_KEY=your_livekit_api_key
  LIVEKIT_API_SECRET=your_livekit_api_secret

• Ensure your LiveKit server is reachable. Example: Soniox + Groq Agent (Python)

# soniox_groq_agent.py
import os
import logging
from dotenv import load_dotenv
from livekit.agents import AgentServer, AgentSession, JobContext, cli, Agent
from livekit.plugins import soniox, groq
from typing import Dict

load_dotenv()
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
logger = logging.getLogger("soniox-groq-agent")

server = AgentServer()

class DiarizationAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="""
                You are a helpful conversational assistant. The user will speak through multiple speakers.
                Transcriptions arrive annotated with speaker tags like <s1>...</s1>.
                Use the speaker tags to address people appropriately in replies.
            """
        )

    async def on_enter(self):
        # Optional warm greeting
        await self.session.generate_reply(instructions="Greet the participants and ask how you can help.")

@server.rtc_session(agent_name="soniox-groq-agent")
async def session_entry(ctx: JobContext):
    <http://logger.info|logger.info>("Starting session for room: %s", ctx.room.name)

    # Configure Soniox STT with diarization enabled
    soniox_stt = soniox.STT(
        params=soniox.STTOptions(
            model="stt-rt-v3",                 # optional: choose model
            enable_language_identification=True,
            language_hints=["en"]
        )
    )

    # Enable diarization and set the output formatting for speaker segments.
    # The format wraps segments in speaker tags like <S1>text</S1>.
    # Note: the LiveKit Soniox plugin exposes a single speaker format option.
    soniox_stt.enable_diarization(
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>"
    )

    # Configure Groq LLM
    groq_llm = groq.LLM(
        model="llama3-8b-8192",
        temperature=0.7,
        parallel_tool_calls=True
    )

    # Optionally add TTS; here we attempt to create a Cartesia TTS if available
    tts_provider = None
    try:
        from livekit.plugins import cartesia
        tts_provider = cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc")
    except Exception:
        <http://logger.info|logger.info>("Cartesia TTS not available or not installed; proceeding without TTS.")

    # Create AgentSession with STT and LLM
    session = AgentSession(
        stt=soniox_stt,
        llm=groq_llm,
        tts=tts_provider,
    )

    # Keep a simple mapping of recent speaker activity to demonstrate speaker identification.
    speaker_activity: Dict[str, int] = {}

    @session.on("user_input_transcribed")
    def handle_transcript(transcript):
        # Only act on final transcripts
        if not transcript.is_final:
            return

        # Soniox diarization provides transcript.speaker_id when enabled
        speaker_id = getattr(transcript, "speaker_id", None) or "unknown"
        text = transcript.text or ""

        # Track simple activity counts (could be sliding window, durations, etc.)
        speaker_activity[speaker_id] = speaker_activity.get(speaker_id, 0) + 1

        # Format the message we will send to the LLM, preserving speaker tag
        # Example: "<S1>Hello, I need help.</S1>"
        formatted_input = f"<{speaker_id}>{text}</{speaker_id}>"

        <http://logger.info|logger.info>("Received final transcript: %s", formatted_input)

        # Create a short prompt that includes speaker context and transcript
        prompt = (
            f"You are a helpful assistant. The conversation includes multiple speakers marked with tags.\n"
            f"Recent transcript: {formatted_input}\n\n"
            f"Respond addressing the speaker appropriately (use their tag) and keep the reply concise."
        )

        # Ask the LLM to generate a reply
        try:
            # Using the session's generate_reply helper integrates with the agent lifecycle
            # We pass the prompt as instructions to generate a context-aware response
            session.llm.generate(prompt)
            # If you prefer to use session.generate_reply (which will create agent messages):
            # await session.generate_reply(instructions=prompt)
        except Exception as e:
            logger.error("LLM generation failed: %s", e)

    # Start the agent and session
    agent = DiarizationAgent()
    await session.start(room=ctx.room, agent=agent)
    await ctx.connect()

if __name__ == "__main__":
    # Validate env
    required = ["SONIOX_API_KEY", "GROQ_API_KEY", "LIVEKIT_URL", "LIVEKIT_API_KEY", "LIVEKIT_API_SECRET"]
    missing = [r for r in required if not os.getenv(r)]
    if missing:
        logger.error("Missing environment variables: %s", ", ".join(missing))
        raise SystemExit(1)

    <http://logger.info|logger.info>("Starting LiveKit agent server...")
    cli.run_app(server)

Notes and explanations • Enabling diarization: ◦

enable_diarization

is invoked by calling

soniox_stt.enable_diarization(...)

and providing a single-format template for speaker segments. The plugin returns speaker IDs (e.g., S1, S2) on final transcripts via

transcript.speaker_id

. • Handling transcripts: ◦ The handler listens to

user_input_transcribed

events and uses

transcript.is_final

to only process stable text. ◦ We preserve speaker tags (e.g.,

<S1>...</S1>

) when sending the snippet to Groq so the LLM sees who spoke. • Simple speaker identification: ◦ The example keeps a basic count of utterances per

speaker_id

in

speaker_activity

. You can replace this with more advanced scoring (time-weighted, speech-duration based) to detect a primary speaker. • LLM integration: ◦ The example shows how to form a prompt including the speaker tag and send it to Groq. Adjust prompt engineering as needed. • TTS: ◦ TTS is optional. If you want spoken replies, configure a TTS plugin and call session methods to speak the LLM response. • Mapping speaker IDs to LiveKit participants: ◦ Soniox provides speaker labels (S1, S2). If you need to map those labels to actual LiveKit participant identities, you must align timestamps of transcription segments with participant track timestamps or inspect per-track audio features to correlate labels to track sources. I can provide a mapping snippet if you want. Next steps I can help with • Provide a runnable example that maps Soniox speaker IDs to LiveKit participant identities using timestamps and track metadata. • Add a sliding-window primary-speaker detector and suppress background speakers (post-processing). • Convert the example to use awaitable LLM calls (async generation) if you prefer full async flow. Which follow-up would you like: mapping speaker IDs to participants, a primary-speaker detector snippet, or a complete runnable demo with synthetic audio? Sources: examples/voice_agents | livekit/livekit_composite | DeepWiki | Soniox STT plugin guide | LiveKit Documentation | Groq LLM plugin guide | LiveKit Documentation | Sarvam TTS plugin guide | LiveKit Documentation | Gemini Realtime Agent with Live Vision | LiveKit Documentation