I'd be happy to guide you through setting up agent-based audio processing to adjust AI-generated audio volume on the server side. You can do this by overriding either the

tts_node

for STT-LLM-TTS pipeline models or the

realtime_audio_output_node

for realtime models. These nodes let you intercept and modify the audio output before it reaches the user, giving you great control over the volume. Here’s a simple overview of how you can implement this: 1. Start by defining a volume attribute in your agent class, usually initialized in the

__init__

method. 2. Override the

tts_node

or

realtime_audio_output_node

to wrap the default audio stream and apply volume scaling. 3. Create an internal method (like

_adjust_volume_in_stream

) that processes each audio frame by adjusting its amplitude according to the current volume level. 4. Convert the audio data to float, scale it, then convert it back to

int16

to avoid clipping while applying the volume multiplier. 5. Optionally, you can expose a tool call (for example,

@function_tool()

) that allows external clients to dynamically change the volume during runtime. Here’s a straightforward example in Python to illustrate this:

from livekit import rtc
from livekit.agents import Agent, ModelSettings
from typing import AsyncIterable
import numpy as np

class VolumeControlAgent(Agent):
    def __init__(self, volume: int = 50) -> None:
        super().__init__(instructions=f"Voice assistant with volume set to {volume}")
        self.volume = volume  # Volume between 0 and 100

    @function_tool()
    async def set_volume(self, volume: int):
        """Allow external control of audio volume."""
        self.volume = max(0, min(volume, 100))  # Clamp between 0 and 100

    async def tts_node(self, text: AsyncIterable[str], model_settings: ModelSettings):
        return self._adjust_volume_in_stream(
            Agent.default.tts_node(self, text, model_settings)
        )

    async def realtime_audio_output_node(
        self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings
    ):
        return self._adjust_volume_in_stream(
            Agent.default.realtime_audio_output_node(self, audio, model_settings)
        )

    async def _adjust_volume_in_stream(
        self, audio: AsyncIterable[rtc.AudioFrame]
    ) -> AsyncIterable[rtc.AudioFrame]:
        stream = None
        async for frame in audio:
            if stream is None:
                stream = utils.audio.AudioByteStream(
                    sample_rate=frame.sample_rate,
                    num_channels=frame.num_channels,
                    samples_per_channel=frame.sample_rate // 10,
                )
            for f in stream.push(frame.data):
                yield self._adjust_volume_in_frame(f)
        if stream is not None:
            for f in stream.flush():
                yield self._adjust_volume_in_frame(f)

    def _adjust_volume_in_frame(self, frame: rtc.AudioFrame) -> rtc.AudioFrame:
        audio_data = np.frombuffer(frame.data, dtype=np.int16)
        audio_float = audio_data.astype(np.float32) / np.iinfo(np.int16).max
        audio_float *= self.volume / 100.0  # Apply volume scaling
        processed = (audio_float * np.iinfo(np.int16).max).astype(np.int16)
        return rtc.AudioFrame(
            data=processed.tobytes(),
            sample_rate=frame.sample_rate,
            num_channels=frame.num_channels,
            samples_per_channel=len(processed) // frame.num_channels,
        )

This method ensures that volume control happens right on the server side, so your AI-generated audio maintains consistent levels regardless of client-side settings. If you have any questions or need a hand with any part of this, please don’t hesitate to ask—I’m here to help! Sources: Agent speech and audio | LiveKit Docs | Agent speech and audio | LiveKit Docs | Pipeline nodes and hooks | LiveKit Docs