Puneet Singh
05/02/2025, 12:33 PM2025/05/02 12:30:45.940 ERROR [HelixHelper] [jersey-server-managed-async-executor-13] Caught exception while updating ideal state for resource: captain_offers_kpi_REALTIME
java.lang.IllegalStateException: Failed to find partition id for segment: captain_offers_kpi_REALTIME_1717632025246_1745973288376_27_e91caabf-bfae-4eb7-a68e-10726ec6e634 of table: captain_offers_kpi_REALTIME
at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:838) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.controller.helix.core.assignment.segment.StrictRealtimeSegmentAssignment.getPartitionId(StrictRealtimeSegmentAssignment.java:145) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.controller.helix.core.assignment.segment.StrictRealtimeSegmentAssignment.assignSegment(StrictRealtimeSegmentAssignment.java:81) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.lambda$assignTableSegment$16(PinotHelixResourceManager.java:2306) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.common.utils.helix.HelixHelper$1.call(HelixHelper.java:126) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.common.utils.helix.HelixHelper$1.call(HelixHelper.java:112) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.common.utils.helix.HelixHelper.updateIdealState(HelixHelper.java:112) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.common.utils.helix.HelixHelper.updateIdealState(HelixHelper.java:240) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.assignTableSegment(PinotHelixResourceManager.java:2298) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
--
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:825) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]Puneet Singh
05/02/2025, 12:36 PM{
"tableName": "captain_offers_kpi",
"tableType": "REALTIME",
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"segmentsConfig": {
"timeColumnName": "created_at",
"replication": "2",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "365",
"segmentPushType": "REFRESH"
},
"tableIndexConfig": {
"loadMode": "MMAP",
"segmentPartitionConfig": {
"columnPartitionMap": {
"captain_id": {
"functionName": "Murmur",
"numPartitions": 12
}
}
},
"streamConfigs": {
"streamType": "kafka",
...
}
},
"ingestionConfig": {
"batchIngestionConfig": {
"segmentIngestionType": "REFRESH",
"batchConfigMaps": [
{
"input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
"input.fs.prop.region": "eu-west-1",
"inputDirURI": "s3://analytics/dev_bi/offers/",
"includeFileNamePattern": "glob:**/*.json",
"inputFormat": "json"
}
]
},
"transformConfigs": [
{
"columnName": "captain_id",
"transformFunction": "captainId"
},
{
"columnName": "customer_car_type_id",
"transformFunction": "customerCarTypeId"
},
{
"columnName": "driver_car_type_id",
"transformFunction": "driverCarTypeId"
},
{
"columnName": "reference_id",
"transformFunction": "referenceId"
},
{
"columnName": "reference_type",
"transformFunction": "referenceType"
},
{
"columnName": "created_at",
"transformFunction": "createdAt"
}
]
},
"metadata": {},
"routing": {
"segmentPrunerTypes": [
"time", "partition"
],
"instanceSelectorType": "strictReplicaGroup",
"useFixedReplica": true
},
"task": {
"taskTypeConfigsMap": {
"UpsertCompactionTask": {
"schedule": "0 0 * * * ?",
"bufferTimePeriod": "5m",
"invalidRecordsThresholdPercent": "30",
"invalidRecordsThresholdCount": "100000",
"tableMaxNumTasks": "40",
"validDocIdsType": "SNAPSHOT"
},
"SegmentGenerationAndPushTask": {
"schedule": "0 0 * * * ?"
}
}
},
"upsertConfig": {
"mode": "FULL",
"metadataTTL": 0,
"deleteRecordColumn": "delete_record",
"dropOutOfOrderRecord": true,
"enableSnapshot": true,
"deletedKeysTTL": 0,
"enablePreload": false,
"consistencyMode": "SNAPSHOT",
"upsertViewRefreshIntervalMs": 3000,
"hashFunction": "NONE",
"defaultPartialUpsertStrategy": "OVERWRITE",
"allowPartialUpsertConsumptionDuringCommit": false
}
}Xiang Fu
tableName__partitionGroupId__sequenceNumber__creationTimePuneet Singh
05/02/2025, 4:46 PMPuneet Singh
05/02/2025, 4:49 PMXiang Fu
strictReplicaGroupPuneet Singh
05/03/2025, 1:27 PMtableName__partitionGroupId__sequenceNumber__creationTime
Any pointers here?
tried using:
segmentNameGeneratorSpec:
type: normalizedDate
configs:
timeColumnName: created_at
timeFormat: '1:MILLISECONDS:EPOCH'
segment.name.prefix: 'captain_kpi_offers_REALTIME'
exclude.sequence.id: falseXiang Fu
UploadedRealtimeSegmentNameGeneratorPuneet Singh
05/03/2025, 3:05 PMXiang Fu
UploadedRealtimeSegmentNameGenerator to create a segment name.Puneet Singh
05/04/2025, 5:41 AM