https://pinot.apache.org/ logo
Join Slack
Powered by
I am ingesting data into an offline pinot table an...
# troubleshooting
p
I am ingesting data into an offline pinot table and noticed that the time to ingest data is increasing with every hourly partition. I also see logs in the spark job's logs like
Copy code
2025-10-02 12:09:58.319	
25/10/02 19:09:58 [dag-scheduler-event-loop] INFO DAGScheduler: Got job 1 (foreach at SparkSegmentMetadataPushJobRunner.java:219) with 2 output partitions
	2025-10-02 12:09:58.319	
25/10/02 19:09:58 [main] INFO SparkContext: Starting job: foreach at SparkSegmentMetadataPushJobRunner.java:219
	2025-10-02 12:09:58.301	
25/10/02 19:09:58 [main] INFO ConsistentDataPushUtils: Consistent data push is: disabled
	2025-10-02 12:09:58.273	
25/10/02 19:09:58 [main] INFO GcsPinotFS: Listed 654 files from URI: <gs://segment-store/pinot/controller-data/lego_gc>, is recursive: true
which makes me believe that it is trying to push all segments and not just the most recent hour's segments to pinot. this is what the batch ingestion spec yaml looks like
Copy code
executionFrameworkSpec:
  name: "spark"
  segmentGenerationJobRunnerClassName: "org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentGenerationJobRunner"
  segmentTarPushJobRunnerClassName: "org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentTarPushJobRunner"
  segmentUriPushJobRunnerClassName: "org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentUriPushJobRunner"
  segmentMetadataPushJobRunnerClassName: "org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentMetadataPushJobRunner"
jobType: "SegmentCreationAndMetadataPush"
inputDirURI: "<gs://input-store/lego_gc/date_key=2025-10-01/hour=00/>"
includeFileNamePattern: "glob:**/*.parquet"
outputDirURI: "<gs://segment-store/pinot/controller-data/lego_gc>"
overwriteOutput: true
pinotFSSpecs:
  - scheme: "gs"
    className: "org.apache.pinot.plugin.filesystem.GcsPinotFS"
pinotClusterSpecs:
  - controllerURI: "pinot.something.com"
recordReaderSpec:
  dataFormat: "PARQUET"
  className: "org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader"
tableSpec:
  tableName: "lego_gc"
  schemaURI: "pinot.something.com/tables/lego_gc/schema"
  tableConfigURI: "pinot.something.com/tables/lego_gc"
segmentNameGeneratorSpec:
  type: "simple"
  configs:
    segment.name.prefix: "lego_gc_2025-10-01-00"
    exclude.sequence.id: true
pushJobSpec:
  pushParallelism: 2
  pushAttempts: 2
  pushRetryIntervalMillis: 1000
do i need to configure outputDirURI to be partitioned as well? like 
gs://segment-store/pinot/controller-data/lego_gc/date_key=2025-10-01/hour=00/
? Please note that 
gs://segment-store/pinot/controller-data/
is the location of segment store as configured in servers.
m
Where’s the time spent?
p
since you are asking that question is it fair to assume that not all segments are getting pushed every time? and only the new ones from current partition are being pushed?
Screenshot 2025-10-02 at 3.35.17 PM.png
m
I suspect that is not the case based on your graph, it might be pushing all each time
p
do i need to configure 
outputDirURI
to be partitioned as well? like 
<gs://segment-store/pinot/controller-data/lego_gc/date_key=2025-10-01/hour=00/>
? Please note that 
<gs://segment-store/pinot/controller-data/>
is the location of segment store as configured in servers. that won't create an issue for servers right? or does segment store needs to have a flat structure, and hence i should use a different location for uploading segments for ingestion?
m
I don’t think so. I say look at the lower level logs from the actual push job
p
i do see an entry in log for every segment being pushed
Copy code
25/10/02 20:37:24 [Executor task launch worker for task 1.0 in stage 1.0 (TID 14)] INFO SegmentPushUtils: Pushing segment: lego_gc_OFFLINE_1758772800000_1758772800000_9 to location: <https://pinot.something.com> for table lego_gc
so instead of 30 segments for an hour, 200+ segments got pushed to pinot
also interesting to note that 
segment.name.prefix: "lego_gc_2025-10-01-00"
is not getting applied to segment file name....
using 
outputDirURI = <gs://segment-store/pinot/controller-data/lego_gc/date_key=2025-10-01/hour=00/>
and so forth reduced the ingestion time for each hour to be ~10-15 mins
also confirmed from the log that only the segments in the partitioned folder are being pushed to pinot instead of all the segments
m
The output dir you use should be outside of Pinots controller data dir.
p
what is the downside of using controller data dir as the output dir?
m
Output of job should be a temp folder that you could simply delete after each run. Also, if you accidentally have wrong setup in your job, you could mess up controller’s data dir, hence messing up the deepstore data.
p
understood. i'll leave this as is for the poc but make sure to use a temp folder in production.
Open in Slack
PreviousNext
Powered by