does anyone happen to know if the minion merge rollup task will leave batch-uploaded segments that conform to the rollup untouched? concrete example: 1. use spark to create parquet files that are bucketed by week with a max of 5 million rows per parquet file 2. ingest those (also via spark) into the offline table, which appears to always create 1:1 input files to segment files a. i do not set

copyToDeepStoreForMetadataPush

to true currently, so the segments are not copied into the

controller.data.dir

3. create a realtime table that is creating segments every 6h 4. add a

RealtimeToOfflineSegmentsTask

to combine and transfer segments from realtime to offline segments (daily buckets) 5. add a

MergeRollupTask

to further compact those segments into weekly buckets i would prefer the

MergeRollupTask

not to go and rewrite every single batch segment i had already uploaded. i assume that’s a fairly common expectation, but it’s not clear from the docs (haven’t dug into the code yet) if it will figure that out correctly and skip the segments that are already aligned, or not?

@Manish ^^

Sorry for the delay. I looked at the code. The only check that skips segments from being picked is if it has already being touched by merge-rollup . So merge-rollup will pick up all segments (even uploaded via batch and meets criteria). Are you concerned about the extra work being done by merge-rollup or that it'll break segment property that you want to rely on for batch uploaded segments ? Merge rollup writes segment metadata with a key

SEGMENT_ZK_METADATA_TIME_KEY

to indicate that it has been picked up merge-rollup and thats seems like the check to skip segments.

thanks for the reply and no worries at all! it’s the former mostly, we are trying to run things as lean as possible (and pinot has really impressed us). rewriting all our data is unideal in that regard. however i think i’ll probably be able to get enough mileage out of the

RealtimeToOfflineSegmentsTask

that perhaps i’ll be able to avoid the

MergeRollupTask

altogether. i’m mostly just trying to get a handle on building optimally-sized segments and keeping things that way over time.