That's why there is a whole process for what people you want to invite for podcasts, you don't invite cold leads, but people already shown intent to buy and know about your brand,
Also podcasts are a great way to build 1-many relations if you use them as community meetups...
Again, as I said don't think of podcasts as an isolated lever, it is part of a very holistic relationship building excercise