Hi guys, I found a bug here when using agent in pi...
# agents
b
Hi guys, I found a bug here when using agent in pipeline mode (non-realtime LLM) + Cartesia TTS (non-English lang): transcript gets spaced per-character and audio degrades. When in realtime mode or using
lang=en
Cartesia, all working fine. Environment • Python • Mode: Pipeline (problem) / Realtime (no problem) • TTS: Cartesia Sonic 3 • STT: OpenAI • LLM: OpenAI / OpenAI Realtime • Language:
ja
(some non-English),
en
(OK) Scope Matrix • Pipeline +
lang=ja
→
❌ Broken • Realtime +
lang=ja
→
✅ OK • Pipeline +
lang=en
→
✅ OK • Realtime +
lang=en
→
✅ OK Description In pipeline mode with
lang=ja
, the transcript renders with a space between every character (e.g., “こ ん に ち は”), and TTS quality degrades after a few turns. Other combinations (realtime or
lang=en
) do not reproduce. Root Cause • Extra space got padded on every chunk in
transcription_node()
• Normally, in
tts_node()
, each sentence were sent into the node as one chunk. After a few turn, each chunk were consist of only one character resulting in the weird speaking voice. Extremely weird in some language like Thai. • https://github.com/livekit/agents/blob/b80088296e79ac496ad268c64bb80818e3742d66/li[…]lugins/livekit-plugins-cartesia/livekit/plugins/cartesia/tts.py • So, I guess the root cause is somewhere around
SynthesizeStream
and
ChunkedStream
Tested with a few more non-English lang • es - ok • ko - ok • th - bug • zh - bug
l
Interesting, and thanks for raising. I have asked someone from our Agents team to take a look.
What llm model are you using? It could be that the model is returning characters with spaces. You can verify what the llm is sending by overriding the LLM node: https://docs.livekit.io/agents/build/nodes/#llm_node As you're probably aware, OpenAI Realtime wouldn't use Cartesia which is probably why those are OK.
b
@lively-winter-88668 Thanks for replying. Let me clarify. > What llm model are you using? 1. Pipeline setup a. stt: openai - gpt-4o-transcribe b. llm_:_ openai - gpt-4o-mini (gpt-4o also yields the bug) c. tts: cartesia - sonic-3 2. Realtime setup a. llm: openai - gpt-realtime with
text
modalities b. tts: cartesia - sonic-3 > You can verify what the llm is sending by overriding the LLM node Ok, will check this out, and update my finding. > As you're probably aware, OpenAI Realtime wouldn't use Cartesia which is probably why those are OK. I used realtime model with the
text
modality, and I can confirm I'm hearing the Cartesia voices. So I think Cartesia is being used here. Please correct me if I misunderstood. . . I've tried juggling different models around in different combinations, and what I can conclude now is
non-realtime LLM
+
some non-english Cartesia
yields the bug.
@lively-winter-88668 I've dig more on lang=
th
since Thai needs a combination of characters to produce a word and pronounce correctly, so this bug is kinda critical for language like this. Here is the result. Setup • llm_:_ openai - gpt-4o-mini (gpt-4o also yields the bug) • tts: cartesia - sonic-3 lang=
th
1. On the first few turns (first image), the LLM outputs looks ok. Each chunk logs a pronounceable word (not perfect but the Cartesia voice pronounces 99% correct). 2. After the first few turns (second image), thing get worse. Each chunk log character by character results in an alien voice from Cartesia.
@lively-winter-88668 I've try setting LLM as
gemini: gemini-2.0-flash-exp
, bug the same.
I might not have any expertise in this, but if there is anything I could help debug/fix, please let me know.
l
Ah, understood about the
text
modality, that makes sense, but feels like Cartesia is a common factor. I can't test this properly as I am only able to speak English, but can you please run https://github.com/livekit-examples/python-agents-examples/blob/main/pipeline-tts/only_greet.py with the Cartesia plugin
Copy code
tts=cartesia.TTS(
                model="sonic-3",
                language="ja",
            )
and modify the
session.say
with some appropriate text and let me know if you hear the alien voice? I would like to isolate this to the Cartesia plugin if possible.
b
Working as expected, the voice is clear and correct.
Also, the transcription has no space in between each character.
Same goes for lang=
ja
.
👀 1
l
I spoke with our engineering team, can you please raise an issue in https://github.com/livekit/agents and provide a minimal code example to reproduce this? (I suggest using the pipeline model) something is going on, and we both see strange behaviour, but it's difficult to reproduce exactly what you are seeing.
b
ok
@lively-winter-88668 Found another key config during preparing the sample code. When _`use_tts_aligned_transcript=True`_ , the bug occurred.
l
b
Saw this from Cartesia docs, maybe no.3 is the exception livekit didn't aware of? @lively-winter-88668
👀 1