yeah, agent transfers work well for this.
I have a translator language as example here, english and vietnamese.
When I ask it to translate to the vietnamese speaker it transfers to the VN agent which has a voice and model that works best for it, and when finished in VNs it transfers back to the english agent. They just keep seamlessly transferring between each other.