One thing I’ve noticed with voice agents is that the issue usually isn’t the voice itself, it’s the conversation state changing mid-call.
If the transcript confidence drops, the AI misreads intent, gets interrupted, or loses context for a second, the TTS can suddenly sound over-energetic or unnatural trying to “recover” the interaction.
A few things that helped me:
• tightening the system prompt tone instructions
• reducing unnecessary enthusiasm words/exclamation patterns
• improving interruption handling
• checking transcripts for misheard phrases before the weird moment happens
• keeping responses shorter during live calls
Most times the strange voice moment actually starts earlier in the pipeline than people expect.