Hi, since I updated to 1.3.3 from 1.2.18 I have an...
# agents
s
Hi, since I updated to 1.3.3 from 1.2.18 I have an issue playing audios with session.say. Strange enough, it works when I do
console
but not on
dev
or
start
and the calling or from the playground. Is there some quick fix even if hacky? This is really blocking me to enjoy the features in the new version, cc @refined-appointment-81829 @lively-winter-88668
this is what I'm using
Copy code
def load_audio(text: str) -> tuple[bytes, int, int, int] | None:
    if text not in TEXT_TO_AUDIO:
        return None
    audio_path = TEXT_TO_AUDIO[text]

    if audio_path.exists() is False:
        logger.error(f"Audio file `{audio_path}` does not exist. Falling back to TTS.")
        return None

    # Load audio using soundfile
    audio_data, sample_rate = sf.read(str(audio_path), dtype="int16")

    num_channels = 1
    samples_per_channel = len(audio_data)

    # Convert to bytes
    frames = audio_data.tobytes()
    return frames, sample_rate, num_channels, samples_per_channel


def build_audio_generator(audio_data: tuple[bytes, int, int, int]):
    """
    Build an async generator that yields a single audio frame from the given audio data.

    Args:
        audio_data: Tuple of (frames, sample_rate, num_channels, samples_per_channel)

    Returns:
        An async generator that yields a single AudioFrame
    """
    frames, sample_rate, num_channels, samples_per_channel = audio_data
    audio_frame = rtc.AudioFrame(
        data=frames,
        sample_rate=sample_rate,
        num_channels=num_channels,
        samples_per_channel=samples_per_channel,
    )

    async def audio_generator():
        yield audio_frame

    return audio_generator
and then
Copy code
audio_data = load_audio(text)
        return await self.session.say(
            audio=build_audio_generator(audio_data)(),
            text=text,
            allow_interruptions=allow_interruptions,
        )
the result is the audio played in playground or phone is way slower, whereas in console plays fine
l
Do you see this issue with https://github.com/livekit-examples/python-agents-examples/blob/main/basics/playing_audio.py ? I don't see any issues with 1.3.3 using that cut down demo which is similar to your code
👀 1
s
yes @lively-winter-88668, that same script when I run with 1.2.18 it sounds fine in the playground, when I run with 1.3.3 it sounds slow
is there some recommended audio features? like sample_rate, num_channels, etc
l
I don't see any specific recommendations, that sample file is 48000 single channel... I'm more confused why you see issues with the file running slowly, whereas I do not. Everything is configured the same, the only possibility I can think of is network conditions?
s
but that would affect 1.2.18 too
any thoughts here @refined-appointment-81829?
not sure if related, but I see that I use livekit 1.0.19, whereas there's a release with 1.9.4. However, pypi has 1.0.19 as the latest. I'm not sure if it's the same package since I see there's also a python-sdks repo, so I'm confused how all this works
a
this could be related to this issue https://github.com/livekit/agents/issues/4039
s
it seems very different since in console it works for me and your video is for console mode. But I hope you're right and they fix both issues in one
FYI https://github.com/livekit/agents/issues/4048, TLDR: solved by recreating the project 🤷‍♂️
l
@square-spoon-72769 🙌 This has been sat in my todo list whilst I was thinking what could be the cause, thanks for the update
👍 1
s
@lively-winter-88668 and @refined-appointment-81829 seeing this again 😞 I still did not find the pattern for this, but doesn't work for me. Sometimes works, sometimes does not. is there any info I can provide to help find the issue? should I post as issue to GH?
hypothesis: enabling "Agent observability" breaks this. Re-opened GH issue: https://github.com/livekit/agents/issues/4048
r
I see, so you disable agent Observability it is fine but you enable it again it acts up?
s
Idk... i'm not super sure to be honest. I've still seen it happening, but less often. Maybe because agent observability restarts the project? could it be related to this?
r
I doubt that causes it. More likely related to audio file bit rate.
s
but the file is always the same. And it's local, I pack it into the docker image
how can it sometimes work and sometimes not?
r
Not sure but if it sounds slow that is most likely bit rate. If it was something else I would expect more static sound.
s
honestly, Idk how to break it, but cannot risk that sometimes does not work. I just tried after 15min. First time was slow, then 10 times of connect / disconnect worked fine ...
could it be this?
Copy code
async def tts_node(
        self, text: AsyncIterable[str], model_settings: ModelSettings
    ) -> AsyncIterable[rtc.AudioFrame]:
        processed_text = self.process_text(text, show_in_log=True)
        return super().tts_node(processed_text, model_settings)  # type: ignore
does this intervene when session.say(audio=...) ?
r
s
I don't use this anymore. Now it's:
Copy code
async def say_audio(self, text: str, allow_interruptions: bool = True):
        """
        Try to play scripted audio for the given text, fall back to TTS if not available.
        """
        audio_path = load_audio(text)
        if audio_path is None:
            return await self.session.say(
                text,
                allow_interruptions=allow_interruptions,
            )
        <http://logger.info|logger.info>(f"Playing scripted audio for text: {text}")
        return await self.session.say(
            audio=audio_frames_from_file(str(audio_path)),
            text=text,
            allow_interruptions=allow_interruptions,
        )
r
Oh, so you rewrote
load_audio
?
s
yeah I didn't know about
audio_frames_from_file
. I saw it somewhere and then updated
load_audio
so just get the path
l
So, I see the update on https://github.com/livekit/agents/issues/4048, it looks like this was related to having Agent Observability enabled, based on Long's comment. That was a good spot, otherwise this would be a very tricky issue to resolve!
s
l
✅ that is the pull request to fix the issue. Looks like it was related to sample rate too, https://github.com/livekit/agents/pull/4098/files.
👍 1