Hello everyone.
Has anyone implemented tone or intent detection in LiveKit to make an agent more empathetic during calls?
Our stack: Deepgram for STT, Gemini Realtime for LLM, and ElevenLabs for TTS.
We want to detect when a user mentions something concerning (e.g., pain, lack of sleep, feeling unwell, emotional stress) from their input speech or transcript, and then dynamically adjust the TTS tone/style to sound more empathetic.
Are there any LiveKit hooks, events, or recommended integration points where we can plug in custom logic to analyze the incoming audio/text and change the TTS behavior in real time?
Anyone doing something similar?