I think another issue here is that metadata push should not need to download at all. I think that is the case when segment is in final location already (controllers data dir). But it should also be the case when it is not (we should fix this).

I’ve got a PR that avoids most of the download (streaming unzip of the two small files in the segment tarball that are used for metadata push), but it didn’t change the overall time as much as I’d expect - went from 3 hours to about 2h 20m for 1200 segments. Though we’re reading from HDFS with pretty good performance, so maybe download wasn’t taking that long. Or the controller is the bottleneck somehow.