can we download segments and put data into csv? @saurabh dubey

While there is no direct way of doing this, if I'd want to try something like this, I'd use the https://github.com/apache/pinot/blob/master/pinot-segment-local/src/main/java/org/[…]not/segment/local/segment/readers/PinotSegmentRecordReader.java

PinotSegmentRecordReader

to do so

PinotSegmentRecordReader segmentRecordReader = new PinotSegmentRecordReader(mySegment);
GenericRow row = new GenericRow();

while (segmentRecordReader.hasNext()) {
    row = segmentRecordReader.next(row);
    <logic to dump row into csv here>
    row.clear();
}

There might be some caveats around sort order, null values etc which you might have to be careful about. The record reader's options allow for those things

Ahh okay

Also cc: @Mayank if there are other ways which I'm missing

My aim is to break one segment into multiple segment so I thought to download and create multiple segment and upload it

I see. I'd rather use a custom minon task for this. It's more or less the reverse of

MergeRollupTaskExecutor

. It should be possible to write and plug your own custom minion task

thanks @saurabh dubey

You can, or you can use the relevant pinot modules as pom dependencies into your other project. But again, for the usecase you described, a custom minion task would be the right way. Also, you should be able to specify the segment size configs at the producer side itself if you want to keep the segment size in check? Is this a realtime table or offline?

okay got it, it is for offline table

Just curious, Why do you want to break the segment into smaller ones? @vishal