Apache Pinot

Hey Team, I was working on batch replacement of multiple segments. I am looking to atomically replace all the segments together. I understand that there is an api for replacing segments but how do i configure the whole flow. we are fine with doubling the storage during the ingestion phase.

There's work that needs to be done for atomic swap

One way would be to version the segments, and only when all segments (old and new) are online in external view, you make the atomic switch.

Broker would have to know how to route query to one version vs other

So far there's only api's. The rest of the work needs to be done

<@U0159CJTTLJ> If you want to replace the segments atomically using the existing replaceSegment API, you need to do the following:

Let’s say that we want to replace the segments like the following:

(s1, s2, s3 ) -&gt; (s4, s5, s6)

1. Call startReplacement: 
• segmentsFrom: s1, s2, s3
• segmentsTo: s4, s5, s6 
  -&gt; The api will return the Id (store this)

2. upload segment s4, s5,s6

3. Call endReplaceSegment
• provide Id that you got from the step 1. 

So, until the step 3 is done, uploaded segment (s4, s5, s6) should not be used.

should not be used or would not be used ? i.e. will pinot handle not using s4,s5,s6 or does the user of the system needs to handle it.