Hi team, I have a question regarding the realtime to offline minion tasks. Our data must keep a certain column sorted throughout segments. In our current flow we do that by waiting for a consuming segment to reach a large enough size (hundreds of MB) and then seal it and let the minion task run nightly and move data to offline. Re-reading the documentation it seems that the

RealtimeToOfflineSegmentsTask

(as opposed to

MergeRollupTask

where this is not mentioned) keeps the data sorted in the segment it builds. If this were the case I would change the configs in order to seal smaller segments more frequently and let the minion bunch all these small segments up and create the larger, sorted segment to the offline table. My question is if there is any way of validating that the data inside of the offline segments does indeed keep the data sorted by the column

Yes you can check metadata.properties file inside segment. There’s an api for that too

I managed to validate it by looking in the metadata.properties. Thank you for that On a similar note I had a question about the inner workings of this job If you set a

maxNumRecordsPerSegment

in the realtime table's config the remainder of records that can't fill a whole segment get sealed in a smaller segment, right ?

Yes the overflow to the next segment

next segment meaning some segment that will be created by the next schedule or will be sealed as a smaller one this schedule ?

Latter