Hi team , Is it possible to backfill ( data from 2018 ) to a parital upsert pinot table , by uploading a segment using controller REST API ? If thats possible, have a followup question on this , as the uploaded segment is read by controller as immutable segment so how does merge in partial upsert work in this case ?

Upsert tables only allow bootstrap, backfill isn’t alllowed afaik. cc: @Kartik Khare @Yupeng Fu

backfill is possible

@Mayank @Yupeng Fu I had a chance to glance over the code with @Qiaochu Liu about this couple of hours back . Looking into the code we were mostly aligned that its not going to work for parital upserts , but not able to concretely confirm the behaviour . So in that case the only backup option is to stream the data if I understand correctly . The major bottle neck here is we have a immutable segment . Just my thought here is if we can convert a immutable to mutuable segment ( not sure if this a valid operation in pinot world ) so we can run partialUpsert.merge () over it , this would solve the problem . This looks working to me if the backfilled segments are filled in-order . Want to hear your thoughts about it ?

I am not sure if the immutable to mutable conversion is possible since metadata is typically inlined and supporting data structures AFAIK would also not work

+1 immutable segment should be immutable.