Apache Pinot

image.png

Hi, another question about realtime2offlinesegmentstask. We found a small bucket time(1h) data would task more than 1 hour to process, especially in mapper phase.
Is it normal? Is there some task configures I’m missing here?

Here is my r2o task config:
“taskTypeConfigsMap”: {
      “RealtimeToOfflineSegmentsTask”: {
        “bucketTimePeriod”: “1h”,
        “bufferTimePeriod”: “6h”,
        “schedule”: “0 0 0/1 * * ?“,
        “roundBucketTimePeriod”: “1m”,
        “mergeType”: “rollup”,
        “value.aggregationType”: “max”,
        “maxNumRecordsPerSegment”: “2000000"
      }
    }

Yes. depends on your data though.
r2o task is *single threaded*. for larger data sets this takes longer time.

What is the segment size being generated for 1h?

Could it be configured to multiple threads?

Is there some suggestion on resource assignment for minion instances?