charlie
07/08/2025, 11:33 PMSegmentTarPush data ingestion jobs. I have segments stored in GCS that I want to upload to my offline table. There was an issue with my table config early on that prevented my RealtimeToOfflineSegmentsTask from working properly. Now the segments have been removed from my realtime table (retention date passed), but haven't been uploaded to the offline table. Because I have a GCS deep store configured, those segments that were removed from my realtime table still exist in GCS. So I want to run the pinot-admin.sh LaunchDataIngestionJob command to upload those segments to my offline table. I have written the following job spec:
executionFrameworkSpec:
name: 'standalone'
segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'
jobType: SegmentTarPush
inputDirURI: 'gs:://my_bucket/my_table_folder'
outputDirURI: 'gs:://my_bucket/my_table_folder'
overwriteOutput: true
pinotFSSpecs:
- scheme: gs
className: org.apache.pinot.plugin.filesystem.GcsPinotFS
tableSpec:
tableName: 'my_table_OFFLINE'
pinotClusterSpecs:
- controllerURI: 'http://my_controller_url.com'
pushJobSpec:
pushAttempts: 2
pushRetryIntervalMillis: 1000
Am I using the right job type for what I'm trying to achieve? Is what I'm trying to do possible?charlie
07/08/2025, 11:35 PMINFO [IngestionJobLauncher] [main] Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMayank
charlie
07/08/2025, 11:42 PM/v2/segments endpoint?Mayank
charlie
07/08/2025, 11:47 PMpinot-admin.sh UploadSegment \\
-controllerHost localhost \\
-controllerPort 9000 \\
-segmentDir /path/to/local/dir \\
-tableName myTablecharlie
07/08/2025, 11:48 PMcharlie
07/08/2025, 11:49 PMIt requires the segment to be stored locally, or that the segment can be opened as an InputStream on PinotFS, so we can stream the entire segment tar file to the controller.Can a segment stored in GCS be opened as an InputStream on PinotFS?
Mayank
Mayank
charlie
07/09/2025, 3:52 PMcharlie
07/16/2025, 11:55 PMexecutionFrameworkSpec:
name: 'standalone'
segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentUriPush
inputDirURI: 'gs://my_bucket/my_dir'
outputDirURI: 'gs://my_bucket/my_table_dir'
overwriteOutput: true
pinotFSSpecs:
- scheme: gs
className: org.apache.pinot.plugin.filesystem.GcsPinotFS
configs:
projectId: 'my-gcp-project-id'
# We use a service account and not a service account key, so I wonder if I can skip this?
# gcpKey: 'path-to-gcp json key file'
tableSpec:
tableName: 'my_table_OFFLINE'
pinotClusterSpecs:
- controllerURI: 'my_controller_url'
pushJobSpec:
segmentUriPrefix: 'gs://my_project/my_dir/my_segment_file'
segmentUriSuffix: ''
pushAttempts: 2
pushRetryIntervalMillis: 1000charlie
07/16/2025, 11:56 PM2025/07/16 23:47:02.221 INFO [IngestionJobLauncher] [main] Trying to create instance for class org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunnerMayank
charlie
07/18/2025, 5:19 PMMayank