I hook up pipecat smart turn model(audio input) to our livekit agents. Any info if are we allowed to do this? seems like a good alternative to transcript based input model.

Allowed? Yes, LiveKit is open source 🙂 but the official line is you shouldn't need to do this as the LiveKit turn detector model is recommended. I can't point you to specific benchmarks comparing the models, but I can point you towards the detection accuracy metrics: https://huggingface.co/livekit/turn-detector#benchmarks

Thanks! I think the biggest issue we ran into with the LiveKit turn detector is that it relies on the transcript for prediction, which makes it highly dependent on transcription quality. In scenarios where the transcriber has a higher error rate especially under non-ideal audio conditions like phone calls the performance degrades significantly. The compounding effect of ASR errors directly impacts turn detection accuracy. Word errors and punctuation generated by the transcriber also biases the model’s predictions. With the Pipecat model, the approach is different since it operates directly on the audio. In my simulations of degraded audio quality, i actually get better results. Anyway its good to hear that we're allowed to use third party models.

I understand you are set-up, but just to add some colour we do also have dedicated noise cancellation for telephony, https://docs.livekit.io/transport/media/enhanced-noise-cancellation/#agents-plugin-available-models, which should improve the transcript quality. STT model choice will also affect this.