I want to change the provider in flight for text t...
# ask-ai
m
I want to change the provider in flight for text to speech. How do I do it for a life kit deployed instance? Any easy way to do it without redeploying?
t
Thanks for reaching out, Rishabh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I understand you want to switch the text-to-speech (TTS) provider for a LiveKit-deployed agent instance without redeploying. Good news: you can do that in-flight. Below are the straightforward options and a recommended approach so you can pick what fits your setup. What I’m responding to • You asked how to change the TTS provider for a LiveKit agent while it’s running, and whether you can do it without redeploying. Recommended approach — Agent handoff (no redeploy) • Create a second Agent configured with the new TTS provider. • Trigger an in-session handoff so the new Agent takes over the session. The handoff preserves chat/session context so conversation history is not lost. • Trigger the handoff programmatically (for example via a function/tool call when an admin or user requests the switch). Why this is the easiest and most reliable • Agent handoffs are designed to swap agent logic (including different plugin/tts configs) at runtime without restarting or redeploying the service. • You keep the session history by passing the existing chat context to the new agent, so the user experience remains continuous. Example flow (conceptual) 1. Define two Agent classes (AgentA uses TTS provider X, AgentB uses TTS provider Y). 2. When you decide to switch, call a handoff function that returns/activates AgentB for the current session. 3. Pass the current session/chat context to AgentB so history persists. 4. The session continues using AgentB’s TTS. Alternative option — Update session/plugin options (if supported) • Some deployments support updating
tts
options on the running session (e.g.,
session.update_options(tts=new_tts_instance)
). • This can work when your runtime exposes an options update API, but support for modifying every plugin field at runtime is not guaranteed. Use handoff if you need a reliable method. Supported providers • You can switch between any TTS providers available to your instance (LiveKit Inference TTS or external plugins like ElevenLabs, Cartesia, etc.). Each provider requires the appropriate plugin setup and credentials. Quick checklist before switching • Ensure the new TTS plugin is installed and configured (API key, voice selection). • Confirm your handoff logic passes session/chat context to the new agent. • Test the handoff in a staging session to validate voice, latency, and context preservation. If you’d like, tell me: • Which TTS provider you’re switching from and to (e.g., Cartesia → ElevenLabs), and • Whether you’re using the LiveKit Agents SDK (Python/JS) or a different integration. I can then give a small code snippet tailored to your environment (agent class example + handoff call) to implement the switch. Sources: TTS Comparison | LiveKit Documentation | Agents and handoffs | LiveKit Documentation | Text-to-speech (TTS) models overview | LiveKit Documentation | Cartesia TTS plugin guide | LiveKit Documentation
m
So basically I have a bunch of credits across multiple text-to-speech providers like Inworld, Carticia and Leavenlabs. I want to automatically roll over to the next one as soon as the credits are close to expiring or have expired. For the said tts service.
t
Thanks — this is a great use case. I can help you design an automated rollover that swaps TTS providers when credits are near expiry or exhausted, and do the switch in-flight using agent handoffs so you don’t need to redeploy. Before I give a concrete design and code snippets, I need a few details so the solution fits your environment: 1. How do you currently track credits for each TTS provider? ◦ Do you have an internal billing dashboard, or do you rely on each provider’s usage/credits API? 1. Do the TTS providers you listed (Inworld, Cartesia, ElevenLabs) expose APIs that return remaining credits, expiration dates, or usage metrics accessible to your service? ◦ If yes, which providers offer what fields (remaining balance, expiry timestamp, usage rate)? 1. Which runtime are you using for LiveKit Agents? ◦ Python, Node/JS, or a different integration? 1. Where should the rollover logic live? ◦ Options: within your LiveKit agent process, a separate orchestration service (cron/job runner), or a serverless function. 1. Desired rollover policy (defaults I can implement if you don’t specify): ◦ Threshold trigger: start switching when remaining credits <= X or expiry within Y days (common default: X = $5 or Y = 3 days). ◦ Fallback order: a fixed priority list (e.g., Inworld → Cartesia → ElevenLabs) or round-robin. ◦ Retry/backoff: how to handle failed provider calls (e.g., retry twice then move to next provider). If you want, I can propose a concrete architecture and provide: • A monitoring pattern that polls/receives webhooks for credit/expiry. • Decision logic (thresholds, priority/fallback). • Example code for LiveKit Agent handoff (Python or JS) that swaps the agent’s
tts
plugin and preserves session history. • Example of a small orchestration service (cron + webhook) that triggers the handoff when thresholds are hit. Tell me your answers to questions 1–4 (or which defaults you prefer) and which language you want the example in, and I’ll produce the tailored design and code snippets.
m
Give me example in Python on how to have the fallback order or retry with a different provider.
t
Thanks — I’ll show a concise, practical Python example you can drop into a LiveKit Agent process to automatically retry and roll over TTS providers when one fails (for example due to exhausted credits or expiry). This uses agent-side logic (no redeploy) to try providers in a priority order and preserve session context. What this answers • How to attempt primary TTS, retry on transient errors, and fall back to the next provider in a configured order. • How to preserve the session and publish audio to the room. Example design choices in the snippet • A configurable ordered list of TTS provider instances (priority list). • Per-provider retry attempts with backoff. • Clear logging and error handling so you can instrument credit-exhaustion detection (e.g., detect specific error types or provider response codes). • Safe fallback without redeploying: the running agent handles the switch. Python example (conceptual; adapt to your runtime and plugin APIs):
Copy code
import asyncio
import logging
from typing import List

from livekit import agents, rtc
from livekit.plugins import elevenlabs, cartesia, inworld  # example plugin imports

logger = logging.getLogger("tts-fallback")
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)

class TTSProviderWrapper:
    def __init__(self, name: str, tts_instance, max_retries: int = 2, retry_delay: float = 1.0):
        self.name = name
        self.tts = tts_instance
        self.max_retries = max_retries
        self.retry_delay = retry_delay

    async def synthesize_stream(self, text: str):
        """
        Attempt to synthesize. Return async iterable/stream of audio or raise.
        Adapt this method to the actual plugin API (some return bytes, some async generators).
        """
        # Example: assume plugin provides async generator `synthesize(text)`
        attempt = 0
        last_exc = None
        while attempt <= self.max_retries:
            try:
                <http://logger.info|logger.info>(f"Trying TTS provider {self.name} (attempt {attempt+1})")
                return self.tts.synthesize(text)  # may be async generator or awaitable
            except Exception as e:
                last_exc = e
                logger.warning(f"TTS {self.name} attempt {attempt+1} failed: {e}")
                attempt += 1
                if attempt <= self.max_retries:
                    await asyncio.sleep(self.retry_delay * attempt)
        raise last_exc

async def publish_audio_stream(room: rtc.Room, audio_stream):
    """
    Publish audio stream to the LiveKit room.
    Adapt depending on audio_stream shape. Here we assume an async iterable yields audio frames/bytes.
    """
    async for chunk in audio_stream:
        await room.local_participant.publish_audio(chunk)

async def speak_with_fallback(
    room: rtc.Room,
    text: str,
    providers: List[TTSProviderWrapper],
):
    last_error = None
    for provider in providers:
        try:
            # Get an async stream/generator for audio from the provider
            audio_stream = await provider.synthesize_stream(text)
            # If synthesize_stream returned an async generator, publish it:
            await publish_audio_stream(room, audio_stream)
            <http://logger.info|logger.info>(f"Succeeded with TTS provider: {provider.name}")
            return provider.name  # success - return which provider was used
        except Exception as e:
            logger.warning(f"Provider {provider.name} failed: {e}")
            last_error = e
            # Optionally detect credit-exhaustion errors here and mark provider as disabled
            # Example:
            # if is_credit_exhaustion_error(e):
            #     disable_provider_in_registry(provider.name)
            continue
    # All providers failed
    logger.error(f"All TTS providers failed. Last error: {last_error}")
    raise last_error

# Example Agent entrypoint
async def agent_entry(ctx: agents.JobContext):
    await ctx.connect()
    room = ctx.room

    # Initialize provider wrappers in the fallback order
    providers = [
        TTSProviderWrapper("Inworld", inworld.TTS(api_key="INWORLD_KEY")),
        TTSProviderWrapper("Cartesia", cartesia.TTS(model="sonic-3", voice="v1")),
        TTSProviderWrapper("ElevenLabs", elevenlabs.TTS(api_key="ELEVEN_KEY")),
    ]

    # Example usage: speak a message, automatically falling back as needed
    try:
        used = await speak_with_fallback(room, "Hello — switching TTS providers automatically.", providers)
        <http://logger.info|logger.info>(f"Spoken using provider: {used}")
    except Exception:
        # handle global failure (log, alert, play silence, etc.)
        logger.exception("Failed to synthesize speech with all providers")

if __name__ == "__main__":
    server = agents.ServiceService()
    # register your agent entrypoint appropriately for your setup
    server.register_agent(agents.JobRequestHandler(entrypoint=agent_entry))
    server.start()
Practical notes and extensions • Detecting credit exhaustion: check for provider-specific error codes/messages. When detected, mark that provider as temporarily disabled (in memory or in a small registry) so future requests skip it until you refresh its status. • Monitoring and refresh: run a background task to poll each provider’s usage/credits API (if available) and re-enable providers when they have credits again. • Order strategy: use a fixed priority list, or compute dynamic ordering (e.g., based on remaining balance, expiry date, latency). • Backoff/retry tuning: adjust
max_retries
and
retry_delay
per provider; some transient network errors should be retried, but quota/credit errors should usually move immediately to the next provider. • Preserve session/context: because the agent process handles the fallback, you keep session/chat context. If you implement agent handoffs instead (creating a new Agent with a different TTS), pass the
chat_ctx
so history is preserved. If you want, I can: • Provide a version that handles provider disable/enable state and a polling loop to check provider balances. • Convert this into a more concrete snippet matching the TTS plugin APIs you use (Inworld/Cartesia/ElevenLabs) — tell me which plugin methods you see in your environment (e.g., whether
synthesize
is an async generator or returns bytes). Sources: Speechify TTS plugin guide | LiveKit Documentation | TTS Comparison | LiveKit Documentation | Introducing LiveKit Inference: A unified model interface for voice AI