I am trying to write my own tts plugin, but my ttfb always shows up as -1. Am I missing something obvious to do in order to fix this?

Thanks for reaching out, Drew. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I am calling push in my SynthesizeStream implementation. Do I need to call flush? Is there something else I could be doing wrong?

Hey there! You're right to ask about

flush()

- it's actually super important in your

SynthesizeStream

setup. Think of it like a crucial checkpoint that does a few key things: • Helps separate each audio segment, just like the framework needs • Makes sure all your metrics get tracked by moving pending text into the collection system • Gives the TTS system a signal to process all waiting text by dropping a

FLUSH_SENTINEL

in the queue If you skip calling

flush()

, things might get a bit wonky - the system could have trouble processing or finalizing your audio data, which might be why that

ttfb

is stubbornly staying at -1. Let me also share some other potential reasons why

ttfb

might be giving you trouble: • No audio frames coming through from the TTS service - double-check that your

push()

is sending those frames correctly • Possible TTS service errors blocking audio generation • Network hiccups interrupting the process • Input text that's empty or not quite right, preventing audio creation • TTS service timeouts happening before any audio shows up • Potential bugs in your TTS plugin stopping audio frames from being created Take a close look at your plugin, and make sure everything's set up to smoothly emit and process those audio frames. You've got this! Sources: agents/src/pipeline/agent_output.ts

I am definitely emitting the audio frames since I can hear my agent just fine, and adding output_emitter.flush() doesn't do anything. Any other ideas?

Hey there! I totally understand you're running into some tricky issues with audio frame emissions and the

ttfb

metric. Let me break down some potential reasons why this might be happening, and hopefully help you troubleshoot. 1. No audio frames reaching the metrics monitoring task: Sometimes frames can get lost in the process - it's like they vanish before the monitoring system can catch them! 2. Error conditions blocking metrics calculation: If something goes wrong during synthesis, the metrics task might just give up and exit before calculating anything. 3. Empty audio generation: Double-check that your audio frames aren't coming up blank. The framework gets pretty strict about this! 4. Timing quirks in streaming mode: Make sure

_mark_started()

is doing its job and

_started_time

isn't stuck at zero. These little details matter. 5. Queue sync headaches: The metrics iterator uses a separate queue, so if anything goes wrong with queueing or the synthesis task, things can get messy. Want to dig deeper? Here are some friendly debugging tips: • Confirm your synthesis task is running smoothly, without any sneaky errors. • Verify those audio frames are actually real and not just silent placeholders. • Check that the metrics monitoring task isn't bailing out early. • In streaming mode, make sure

_mark_started()

gets called right when you start pushing text. Hope this helps you track down what's going on! You've got this. 😊 Sources: plugins/resemble/src/tts.ts | livekit-agents/livekit/agents/tts/tts.py

I have confirmed that self._tts.emit("metrics_collected", metrics) is getting called with the correct ttfb in async def _metrics_monitor_task(self, event_aiter: AsyncIterable[SynthesizedAudio]) -> None: of SynthesizeStream, but I never get the logger info of the TTS metrics, and ttfb shows as -1. What could be going on?

Hey there! I totally understand the frustration of metrics not showing up when they should. Let me help you walk through some potential reasons why this might be happening. 1. Error state prevention: Sometimes, if

self._current_attempt_has_error

is set, your metrics monitor task might be bowing out early. It's like hitting the emergency brake before collecting those important stats. 2. Streaming state issues: For streaming TTS, double-check that

_mark_started()

is called and

_started_time

is set up correctly. If these are missing, your metrics are basically playing hide and seek! 3. Adapter overrides: Some TTS adapters can be a bit sneaky and disable metrics collection by completely overriding

_metrics_monitor_task

. Crafty, right? 4. Event iteration issues: Your metrics monitor task relies on smoothly moving through

SynthesizedAudio

events. If this journey gets interrupted, those metrics won't make it to the finish line. 5. Missing event listeners: Make sure you've got your event listeners properly set up. Here's a quick example:

@session.on("metrics_collected")
def _on_metrics_collected(ev: MetricsCollectedEvent):
    metrics.log_metrics(ev.metrics)

Let's do some detective work and troubleshoot: • Is

_current_attempt_has_error

being set when it shouldn't be? • Are you calling

_mark_started()

for streaming TTS? • Could your adapters be blocking metrics collection? • Are your

"metrics_collected"

event listeners all set up? • Is your metrics monitor task completing its event journey smoothly? I'm confident these checks will help you track down those elusive metrics. You've got this! Sources: livekit-agents/livekit/agents/stt/stt.py | Capturing metrics | LiveKit Docs

It seems like they are not getting emitted to the room

Hey there! Looks like you're having some trouble getting metrics to show up in the room. No worries - let's break down what might be going on. 1. Error state prevention: Sometimes, if

_current_attempt_has_error

is

True

, the metrics monitor task will bail out early. Think of it like a safety switch that stops things when something goes wrong during synthesis. 2. Task lifecycle issues: Your metrics task might be getting cut short, or the

event_aiter

might not be cooperating - basically, it's not collecting or passing along the metrics like it should. 3. Missing event listeners: Metrics need someone to listen for them! If no one's registered to catch the

"metrics_collected"

event, they'll just slip by unnoticed. Here's what that might look like:

this.#tts.on(TTSEvent.METRICS_COLLECTED, (metrics) => {
  if (!speechData) return;
  this.emit(VPAEvent.METRICS_COLLECTED, { ...metrics, sequenceId: speechData.sequenceId });
});

1. Session or connection issues: Even if your TTS component is emitting metrics, they still need to make it through the agent session to reach the room. It's like a relay race where the baton might get dropped. 2. Streaming vs non-streaming differences: Heads up - streaming TTS emits metrics per segment when

ev.is_final

is true, while non-streaming TTS does it all at once at the end. Make sure you're matching the right approach for what you're doing. Let's do some detective work and check: • Is

_current_attempt_has_error

accidentally getting set to

True

? • Is the metrics task actually finishing up properly? • Are your event listeners set up correctly for

"metrics_collected"

events? • Is the agent session passing metrics to the room without any hiccups? These steps should help you track down why those metrics are playing hide and seek. You've got this! Sources: livekit-agents/livekit/agents/stt/stt.py | agents/src/tts/stream_adapter.ts

Could it be that my plugin is not correctly being registered?

Hey there! I totally understand the frustration of plugin registration issues. Let me walk you through some common reasons why your plugin might not be registering correctly, and help you get things back on track. 1. Missing or incorrect plugin registration call: No worries! Just double-check that you're calling

Plugin.register_plugin()

in your

__init__.py

file. Here's a quick example to help you out:

from livekit.agents import Plugin

class CustomTTSPlugin(Plugin):
    def __init__(self) -> None:
        super().__init__(__name__, __version__, __package__)

Plugin.register_plugin(CustomTTSPlugin())

1. Incorrect plugin class implementation: It's an easy fix! Make sure you're calling the parent constructor with the right parameters. Something like this:

super().__init__(__name__, __version__, __package__, logger)

1. Package structure problems: Let's make sure you've got all the right files in place: •

__init__.py

•

version.py

•

pyproject.toml

•

README.md

•

implementation.py

1. Missing TTS interface implementation: Your plugin needs to have the right TTS interface. For TTS plugins, you'll want to create a

TTS

class that inherits from

tts.TTS

. 2. Incorrect exports: Don't forget to properly export your TTS class and other components in the

__all__

list:

__all__ = ["TTS", "ChunkedStream", "__version__"]

1. Import and module issues: Just a quick check to make sure you're importing everything correctly:

from livekit.agents import Plugin

1. Missing version information: Pro tip: Include a

__version__

variable, usually imported from a

version.py

file. 2. Error handling: Keep your error messages clear and handle exceptions gracefully to avoid any registration hiccups. I hope this helps! Take a moment to go through these steps, and I'm confident you'll get your plugin up and running smoothly. If you're still stuck, don't hesitate to reach out – we're here to help! Sources: livekit-plugins/livekit-plugins-baseten/livekit/plugins/baseten/tts.py | Plugin System | livekit/livekit_composite | DeepWiki

does my plugin have to be defined in livekit.plugins?

Hey there! I totally get how plugin registration can feel a bit tricky. Let me break down how to get your custom TTS plugin working smoothly with LiveKit. Think of it like introducing a new friend to your existing group – there are just a few simple steps to make sure everyone gets along! 1. Import the Plugin base class:

from livekit.agents import Plugin

1. Create a custom plugin class: It's like giving your plugin its own identity card – you'll want to inherit from

Plugin

and set up its basic info:

class CustomTTSPlugin(Plugin):
    def __init__(self) -> None:
        super().__init__(__name__, __version__, __package__)

1. Register the plugin: This is basically saying "Hey LiveKit, meet my new plugin!" Just use `Plugin.register_plugin()`:

Plugin.register_plugin(CustomTTSPlugin())

Pro tip: Make sure your plugin fits right into the LiveKit ecosystem. It's like finding the perfect spot in a cozy friend group! Sources: livekit-plugins/livekit-plugins-speechify/livekit/plugins/speechify/tts.py | Plugin System | livekit/livekit_composite | DeepWiki

Is there a bug with the fallback adapter that causes ttfb to be reported incorrectly?

Hey there! I noticed something important about LiveKit's fallback adapter that might affect your metrics tracking. Basically, when this adapter is in use, it turns off metrics monitoring – those tracking methods are just placeholders that don't do anything real. What does this mean for you? If you're using the fallback adapter, just know that

ttfb

metrics won't be reported. No worries, though! If tracking those metrics is crucial for you, you'll want to switch to a non-fallback adapter. I'm happy to help you figure out the best solution for your specific setup. 😊 Sources: livekit-agents/livekit/agents/llm/fallback_adapter.py | TTSMetrics | LiveKit Agents