The crux of the downsides, for me, are that mistakes make it too far down the line. The developer has the power to stop bugs before they happen. When someone sees that they have a safety net in the form of a QA group (that can take days or even weeks to provide feedback) they worry less about the impact of their code. Everything slows down, and avoidable mistakes start showing up everywhere.