So, the way it works is that the connections always start through DERP, but then once they discover each others IPs (direct and NAT'ed), they can attempt to connect directly.