Have you tried the Krisp background voice cancellation? It can do a good job of filtering out background, but the processed audio sometimes causes more errors in the STT than just sending in the unprocessed audio to the STT. but it's worth trying for your use case.
https://docs.livekit.io/home/cloud/noise-cancellation/#overview