https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, I am trying to backfill segments in Pinot...
# troubleshooting
a
Hi team, I am trying to backfill segments in Pinot batch ingestion, but instead of replacing existing segments, Pinot creates new segments with different names (including sequence numbers and UUIDs), resulting in duplicate data for the same time periods. For example, I have two segments for the same date (2025-07-15): 
req_lost_metadata_push_OFFLINE_2025-07-15_2025-07-15_0_3cf9cf26-a02a-4ac8-815a-1d1f2337bc12
and 
req_lost_metadata_push_OFFLINE_2025-07-15_2025-07-15_302_6a2eba3e-f266-4f6a-8735-24345cb795aa.
This is the config i am using-
Copy code
"task": {
      "taskTypeConfigsMap": {
        "SegmentGenerationAndPushTask": {
          "schedule": "0 0 2 * * ?",
          "inputDirURI": "<s3://data-pinot-dev/pinot/input/>",
          "includeFileNamePattern": "glob:**/databackfill",
          "excludeFileNamePattern": "glob:**/_*",
          "inputFormat": "parquet",
          "push.mode": "TAR",
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "ap-southeast-1",
          "segmentNameGenerator.type": "normalizedDate",
          "segment.name.prefix": "req_lost_metadata_push",
          "exclude.sequence.id": true,
          "append.uuid.to.segment.name": false,
          "use.global.directory.sequence.id": false
        }
      }
    }
m
Did you specify table as APPEND or REFRESH?
s
m
Thanks @Shounak Kulkarni . So multiple segments per day is because of multiple input files or there’s a row / size threshold
s
@Anshu Roy these configs needs to be configured under 
batchConfigMaps
and not 
taskTypeConfigsMap
Copy code
"exclude.sequence.id": "true",
"append.uuid.to.segment.name": "false",
"use.global.directory.sequence.id": "false"
let's cleanup the table config to avoid further confusion. Keep the task configs limited to
Copy code
"task": {
      "taskTypeConfigsMap": {
        "SegmentGenerationAndPushTask": {
          "schedule": "0 0 2 * * ?"
        }
      }
    }
an put all the configs under batch configs
Copy code
"batchIngestionConfig": {
        "batchConfigMaps": [
          {
            "inputDirURI": "<s3://data-pinot-dev/pinot/input/>",
            "outputDirURI": "<s3://data-pinot-dev/pinot/segments/>",
            "includeFileNamePattern": "glob:**/databackfill",
            "excludeFileNamePattern": "glob:**/_*",
            "inputFormat": "parquet",
            "push.mode": "METADATA",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader",
            "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
            "input.fs.prop.region": "ap-south-1",
            "segmentNameGenerator.type": "normalizedDate",
            "segment.name.prefix": "req_lost_metadata_push",
            "append.uuid.to.segment.name": "false"
          }
        ],
        "segmentIngestionType": "APPEND",
        "segmentIngestionFrequency": "DAILY",
        "consistentDataPush": false
      }
a
Alright , thank you so much!
This is the final table config i am using , also switched to tar based push. With this config also backfill creates new segments instead of replacing old.
Copy code
{
  "OFFLINE": {
    "tableName": "req_lost_tar_push_OFFLINE",
    "tableType": "OFFLINE",
    "segmentsConfig": {
      "schemaName": "req_lost_tar_push",
      "replication": "1",
      "timeType": "DAYS",
      "timeColumnName": "DT",
      "minimizeDataMovement": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [
        "CITY",
        "CATEGORY",
        "SUPERCATEGORY",
        "CLUSTER",
        "COUNTRY"
      ],
      "aggregateMetrics": false,
      "starTreeIndexConfigs": [
        {
          "dimensionsSplitOrder": [
            "COUNTRY",
            "CLUSTER",
            "SUPERCATEGORY",
            "CATEGORY",
            "CITY",
            "DT"
          ],
          "functionColumnPairs": [
            "SUM__SERVICEABLE_REQUEST_LOST",
            "SUM__GROSS_REQUEST_COUNT"
          ],
          "maxLeafRecords": 1000
        }
      ],
      "enableDefaultStarTree": false,
      "nullHandlingEnabled": false,
      "rangeIndexColumns": [
        "DT"
      ],
      "rangeIndexVersion": 2
    },
    "task": {
      "taskTypeConfigsMap": {
        "SegmentGenerationAndPushTask": {
          "schedule": "0 0 2 * * ?"
        }
      }
    },
    "ingestionConfig": {
      "batchIngestionConfig": {
        "batchConfigMaps": [
          {
            "inputDirURI": "<s3://data-pinot-dev/pinot/input/req_lost_metadata_push2_OFFLINE/>",
            "includeFileNamePattern": "glob:**/*data",
            "excludeFileNamePattern": "glob:**/_*",
            "inputFormat": "parquet",
            "push.mode": "TAR",
            "recordReader.className": "org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader",
            "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
            "input.fs.prop.region": "ap-southeast-1",
            "segmentNameGenerator.type": "normalizedDate",
            "segment.name.prefix": "req_lost_tar_push",
            "exclude.sequence.id": "true",
            "append.uuid.to.segment.name": "false"
          }
        ],
        "segmentIngestionType": "APPEND",
        "segmentIngestionFrequency": "DAILY",
        "consistentDataPush": false
      },
      "continueOnError": false,
      "rowTimeValueCheck": false,
      "segmentTimeValueCheck": true
    },
    "isDimTable": false
  }
}
Executing SegmentGenerationAndPushTask with task config: Task Type: SegmentGenerationAndPushTask, Configs: {segment.name.prefix=req_lost_tar_push, <http://append.uuid.to|append.uuid.to>.segment.name=true, segmentNameGenerator.type=normalizedDate, inputDirURI=s3:
append.uuid.to.segment.name -- This config is not being picked
Copy code
req_lost_tar_push_OFFLINE_2025-08-16_2025-08-16_0_caeec64f-a823-4cba-b0ee-7af9bf7c1428	

req_lost_tar_push_OFFLINE_2025-08-16_2025-08-16_334_4ac37f61-8fe1-4146-b5db-f8c33a7c82ea
found a similar case here - https://apache-pinot.slack.com/archives/C011C9JHN7R/p1738958776476119?thread_ts=1738939734.481829&amp;cid=C011C9JHN7R @Shounak Kulkarni Is it possible to override this config?
Any other way which can be used with minion based ingestion to replace segments?
s
Yeah, Mayank pointed out the other thread to me as well, looks like we cannot override it. I am yet to test the consistent push with minions, will try out and let you know. Just to get more idea about your usecase, how are you backfilling the data? are you updating the same file at source or creating a new file with intended data?
a
We are creating a new file and pushing it to s3 , from s3 we have set up minion based ingestion. So to backfill for same date we are maintaining that files in a backfill folder. Table Folder- normal run backfill run
s
Ok, so anyways consistent push option is out of picture as you have APPEND table. I don't think this is achievable out of the box from the minion task route.
a
Alright, thanks!
s
We can explore the route of leveraging the 
startReplaceSegments
and 
endReplaceSegments
endpoints ref (which consistent push uses underneath) and have segments generation and push outside minion if the backfill is not a frequent operation.
a
Actually we have to replace segments daily , our data source builds data for last 3 days. So last 1 day data is fresh and for other 2 days we will have to replace. So this api can choke the controller if many requests come at once? We were thinking to delete segments based on date , any better way available to do this?
Also for refresh tables whole data is replaced daily , right? This is not on segment basis? `REFRESH`: Entire data is replaced every time during a data push. Refresh tables have no retention.
s
if its needed daily then it makes sense to create a 
REFRESH
table, but I am not sure if that works out of the box with the minion task as well. BTW what's the table size/number of segments?
a
Around 1.5 GB , 365 segments ( 1 segment per day) Refresh is not supported with minion even with full refresh or segment wise refresh is not supported?
s
then you can create a 
REFRESH
table and use consistent push https://docs.pinot.apache.org/operators/operating-pinot/consistent-push-and-rollback
a
It's not explicitly mentioned , it is supported with minion based ingestion , is it supported?
s
No its not, needs to be externally triggered
a
@Shounak Kulkarni Just to confirm this - will this replacement be supported if i use DataIngestionJobs (job spec + pinot-admin LaunchDataIngestionJob) instead of minion based ingestion . Can i have a config like this there -
Copy code
"segmentNameGenerator.type": "normalizedDate",
            "segment.name.prefix": "req_lost_tar_push",
            "exclude.sequence.id": "true",
            "append.uuid.to.segment.name": "false"
and uuid wont be added here , and same name segments would be replaced?
s
Once you enable the consistent push, you need not worry about the segment names, the segments push job runner will take care of swapping the queryable segments at the end of segment push operation completion.
just make sure the 
segmentIngestionType
is 
REFRESH
and 
consistentDataPush
is 
true
a
But for refresh whole table is refreshed daily , right? If i want an append table, because there are tables which have Gbs of data.
s
If i want an append table, because there are tables which have Gbs of data.
Are you changing subset of data for these tables or eveything like the above table?
a
Actually we have to replace segments daily , our data source builds data for last 3 days. So last 1 day data is fresh and for other 2 days we will have to replace.
We need to change subset of data.
Also Consistent Push and rollback is supported only for standalone , we cant do with spark jobs right?
s
> Also Consistent Push and rollback is supported only for standalone , we cant do with spark jobs right? I see 
SparkSegmentMetadataPushJobRunner
handles the consistent push as well. > is this possible Yes it should honour the provided configs for standalone run tldr : Use consistent push if you are generating/replacing all segments on a table to ensure query consistency. Use standalone run to replace subset of segments as minion task force includes uuid.
a
Thank you so much!
any doc available for - 
SparkSegmentMetadataPushJobRunner
?
s
this is more detailed
a
see 
SparkSegmentMetadataPushJobRunner
handles the consistent push as well -- any explicit config which needs to added for this ? as the doc does not have this
s
just add 
"consistentDataPush" : "true"
in batch ingestion config
a
Alright , thanks!
🥂 1
s
Also make sure you use spark3 , 2.4 does not support consistent push
thanks 1
👍 1
Open in Slack
PreviousNext
Powered by