Got it.. i like your idea.. another option is it to make the push job or the controller code to be smarter? If no change in crc, then skip the segment

I was curious about that - if I send a metadata push to the controller, where the segment CRC is the same as what was previously pushed, how efficient is that happy path through the code? E.g. does it avoid any more downloading? I’m assuming the segment CRC is stored in Zookeeper, so this short circuit is already happening, but I haven’t checked to confirm.

It avoids downloading on the server side but currently controller is not efficient

We have a PR out for review already where it avoids overwriting existing segments if a segment with the same name already exists. Will this work for your usecase, @User ? cc: @User

@User I tagged you on PR explaining our rationale. Not sure if this solves your problem. We will need this regardless (a bit time critical for our use). Thanks

@User @User - it looks like your PR triggers a

ControllerApplicationException

if a segment with the same name exists, and

allowRefresh

is false, yes?

That’s right.

I’ve opened a PR for filtering segments during a push - see https://github.com/apache/pinot/pull/8191