Anyone having any new issues with the AI interrupt...
# agents
m
Anyone having any new issues with the AI interrupting the user? We haven't changed anything and randomly started having the AI be incredibly eager to interrupt users and start speaking It seems like the EOU is overeagerly predicting that a user has finished speaking when they haven't. A response generates and then the response cuts itself off within a couple hundred milliseconds, but it messes up the user
Here's an example audio file of what I'm talking about:
c
Are you sing Deepgram FLux? I have had the same issue
m
We're using Nova-3. I was holding off on Flux because I had been hearing of issues like this
c
I got issue with Deepgram in general. I temporarily switched to Assembly and seems to do better but have not had enough time to really gauge. I am excited about Eleven’s Scribe Realtime so I hope Livekit can integrate them soon
🙌 2
m
Yeah maybe i'll try out Assembly and Open Ai to see if those have similar issues or not.
c
@melodic-arm-17634 lmk your thoughts on if Assembly or OpenAI have solved that issue. The downside I’ve noticed is that latency is much higher..
m
We have faced some issues recently too. One of the key thing is to have the right turn detection startegy in place. We have got some success but stil doing some RnD. @cuddly-cartoon-47334 @melodic-arm-17634 have you done any thing specific along those lines ?
c
When using Flux I removed the turn detection model and VAD and tried to use Flux natively. Assembly + turn detection + VAD is too slow for my use case
I tried increasing the eager EOT for flux and seems better but yeah not sure… need to get more data
m
We ended up switching to use
turn_detection="stt"
when configuring the agent session last night, and as of right now it seems like that's working a lot better. We're using nova-3 for stt
c
I thought nova 3 didn’t have its own turn detection and that’s why they built flux? Setting that would just disable VAD right?
m
I didn't either - I'm honestly not completely sure what's happening; it might be VAD only but it seems to be interrupting less that the LiveKit turn detection model was doing I saw this segment: https://docs.livekit.io/agents/build/turns/#stt-endpointing and tried it with our current stt (nova-3) and it worked without complaining at all
c
Huh that’s odd. Anyways, I switched from AssemblyAI to Flux with 0.7 as the eager end of turn and it no longer interrupts and is quite responsive for me. Will have to give it more time in production to see how it’s doing.
m
Yeah let me know what you think! I’d love to switch to flux Or possibly 11 labs scribe V2 when that’s ready
r
Were you using default settings for TURN/VAD in the example you recorded above?
m
@refined-appointment-81829 VAD was completely default
min_endpointing_delay
was set to 1.6 to try to make it interrupt less everything else end of turn was default
One thing I noticed in testing this (by playing the audio file from a separate source while connected to an agent) was that it seemed like DeepGram wasn't transcribing some of the filler words that took a second or two. Maybe End of Turn saw the end of phrase (due to no filler words) and started the response generation during that time? But even in that case, VAD should have blocked that from happening because there was voice activity, correct?
@cuddly-cartoon-47334 still having good luck with Flux? Have you tried scribe v2 yet?
c
I haven’t tried scribe v2 yet is it out on Livekit? Flux seems ok so far
m
Yeah i don't know exactly what version, but it was merged into main: https://github.com/livekit/agents/issues/3881
c
I tried the scribe v2 but its too slow. I think from all the testing Ive done I determined deepgram flux is the best atm
lmk if u find otherwise tho
@melodic-arm-17634
m
I usually wait to hear some community reactions before testing it myself, so I haven't tried scribe yet