https://livekit.io logo
Join Slack
Powered by
Is there a way to bypass TTS response by caching a...
# ask-ai
m
Is there a way to bypass TTS response by caching an audio? mainly to avoid using a lot of tokens while testing.
t
Thanks for reaching out, Rishabh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — great question. You're asking whether you can bypass TTS during testing by playing cached audio to avoid using TTS tokens. Yes — you can do that with LiveKit Agents by supplying pre-synthesized audio to 
session.say()
instead of sending text for synthesis. What this gives you • Completely skips the TTS model (no TTS tokens used). • Plays any pre-recorded audio (.wav/.mp3) wrapped as audio frames. • Keeps session events/transcripts working if you still pass optional transcript text. • Lets you switch between cached audio in tests and real TTS in production. How to do it (high-level) 1. Pre-synthesize and store your audio files (e.g., 
.wav
) for the responses you want to reuse. 2. When testing, read the audio file and stream it to 
session.say()
as an audio generator/stream instead of passing 
text
. 3. Optionally include 
text
for transcripts or fallback purposes. Examples Python (idea):
Copy code
import wave
from livekit import rtc

def audio_generator(file_path):
    with wave.open(file_path, 'rb') as wav:
        frames = wav.readframes(wav.getnframes())
        audio_frame = rtc.AudioFrame(
            data=frames,
            sample_rate=wav.getframerate(),
            num_channels=wav.getnchannels(),
            samples_per_channel=wav.getnframes()
        )
        yield audio_frame

# Use cached audio to avoid TTS
await session.say(audio=audio_generator("cached_response.wav"), text="Optional transcript text")
TypeScript (idea):
Copy code
// Create a ReadableStream<AudioFrame> from a cached audio file and pass it:
await session.say('Optional transcript text', {
  audio: audioStream, // ReadableStream<audioframe>
});
Testing vs production • In test mode: call 
session.say(audio=...)
or pass an audio stream. • In production: call 
session.say(text=...)
to use live TTS. If you'd like, I can: • Show a full example for your runtime (Node/Python) that converts a .wav/.mp3 to audio frames/stream compatible with 
session.say()
. • Suggest a simple test toggle pattern to switch between cached audio and TTS. Which language/runtime are you using for your Agent so I can provide a concrete snippet? Sources: Agent speech and audio | LiveKit Documentation
m
python.
t
Great — thanks for confirming Python. Below is a concise, practical example showing how to play cached audio with 
session.say()
to bypass TTS during testing, plus a simple toggle to switch between test and production behavior. Notes: • This streams a pre-synthesized WAV file as audio frames so no TTS synthesis or tokens are used. • Optionally pass 
text
for transcripts/events while still using cached audio. • Adjust the WAV reading logic if your files are stereo/mono or different sample formats. Example (Python):
Copy code
import wave
import asyncio
from typing import Iterator

# Replace with the actual LiveKit session import/path in your project
# from livekit import rtc

class DummyAudioFrame:
    # Replace with livekit.rtc.AudioFrame in real code
    def __init__(self, data: bytes, sample_rate: int, num_channels: int, samples_per_channel: int):
        self.data = data
        self.sample_rate = sample_rate
        self.num_channels = num_channels
        self.samples_per_channel = samples_per_channel

def wav_to_audio_frames(path: str) -> Iterator[DummyAudioFrame]:
    """
    Read a .wav file and yield a single AudioFrame (or multiple frames if you chunk).
    Replace DummyAudioFrame with livekit.rtc.AudioFrame in your code.
    """
    with wave.open(path, "rb") as wav:
        frames = wav.readframes(wav.getnframes())
        sample_rate = wav.getframerate()
        num_channels = wav.getnchannels()
        samples_per_channel = wav.getnframes()
        # Yield one frame containing the whole file. You can chunk this for long files.
        yield DummyAudioFrame(
            data=frames,
            sample_rate=sample_rate,
            num_channels=num_channels,
            samples_per_channel=samples_per_channel,
        )

async def say_cached_audio(session, wav_path: str, transcript_text: str = ""):
    """
    Play cached audio via session.say().
    `session` is your LiveKit Agent session object.
    """
    audio_iter = wav_to_audio_frames(wav_path)

    # In real code, pass the audio generator/stream compatible with LiveKit's API.
    # Example signature: await session.say(audio=audio_iter, text=transcript_text)
    await session.say(audio=audio_iter, text=transcript_text)

async def say_with_tts(session, text: str):
    """
    Use normal TTS path (this uses tokens).
    """
    await session.say(text=text)

# Toggle between test (cached audio) and production (TTS)
async def respond(session, text: str, cached_wav: str = None, use_cached: bool = True):
    if use_cached and cached_wav:
        # Play cached audio, pass text optionally for transcripts
        await say_cached_audio(session, cached_wav, transcript_text=text)
    else:
        # Use TTS
        await say_with_tts(session, text)

# Example usage (within your async agent handler):
# await respond(session, "Hello, this is a test response", cached_wav="responses/hello.wav", use_cached=True)
If you'd like, I can: • Provide the exact import and 
AudioFrame
usage for your LiveKit Python package (share the package name/version if different), • Show how to chunk larger WAV files into multiple frames for streaming, • Or give a small script that converts MP3 files to WAV and adjusts sample rates if needed. Which of those would be most helpful?
Open in Slack
PreviousNext
Powered by