Hi Team, I am facing below error while trying to ...
# troubleshooting
p
Hi Team, I am facing below error while trying to bootstrap data into Pinot table. What could be the possible cause? Configs in the thread:
Copy code
2025/05/02 12:30:45.940 ERROR [HelixHelper] [jersey-server-managed-async-executor-13] Caught exception while updating ideal state for resource: captain_offers_kpi_REALTIME
java.lang.IllegalStateException: Failed to find partition id for segment: captain_offers_kpi_REALTIME_1717632025246_1745973288376_27_e91caabf-bfae-4eb7-a68e-10726ec6e634 of table: captain_offers_kpi_REALTIME
	at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:838) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.controller.helix.core.assignment.segment.StrictRealtimeSegmentAssignment.getPartitionId(StrictRealtimeSegmentAssignment.java:145) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.controller.helix.core.assignment.segment.StrictRealtimeSegmentAssignment.assignSegment(StrictRealtimeSegmentAssignment.java:81) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.lambda$assignTableSegment$16(PinotHelixResourceManager.java:2306) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.common.utils.helix.HelixHelper$1.call(HelixHelper.java:126) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.common.utils.helix.HelixHelper$1.call(HelixHelper.java:112) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.common.utils.helix.HelixHelper.updateIdealState(HelixHelper.java:112) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.common.utils.helix.HelixHelper.updateIdealState(HelixHelper.java:240) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.assignTableSegment(PinotHelixResourceManager.java:2298) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
--
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:825) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-be7cbbc4ac08bd2ee7ecc6364f75a0199ac83a80]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Copy code
{
  "tableName": "captain_offers_kpi",
  "tableType": "REALTIME",
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "segmentsConfig": {
    "timeColumnName": "created_at",
    "replication": "2",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "365",
    "segmentPushType": "REFRESH"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "captain_id": {
          "functionName": "Murmur",
          "numPartitions": 12
        }
      }
    },
    "streamConfigs": {
      "streamType": "kafka",
      ...
    }
  },
  "ingestionConfig": {
    "batchIngestionConfig": {
      "segmentIngestionType": "REFRESH",
      "batchConfigMaps": [
        {
          "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
          "input.fs.prop.region": "eu-west-1",
          "inputDirURI": "s3://analytics/dev_bi/offers/",
          "includeFileNamePattern": "glob:**/*.json",
          "inputFormat": "json"
        }
      ]
    },
    "transformConfigs": [
      {
        "columnName": "captain_id",
        "transformFunction": "captainId"
      },
      {
        "columnName": "customer_car_type_id",
        "transformFunction": "customerCarTypeId"
      },
      {
        "columnName": "driver_car_type_id",
        "transformFunction": "driverCarTypeId"
      },
      {
        "columnName": "reference_id",
        "transformFunction": "referenceId"
      },
      {
        "columnName": "reference_type",
        "transformFunction": "referenceType"
      },
      {
        "columnName": "created_at",
        "transformFunction": "createdAt"
      }
    ]
  },
  "metadata": {},
  "routing": {
    "segmentPrunerTypes": [
      "time", "partition"
    ],
    "instanceSelectorType": "strictReplicaGroup",
    "useFixedReplica": true
  },
  "task": {
    "taskTypeConfigsMap": {
      "UpsertCompactionTask": {
        "schedule": "0 0 * * * ?",
        "bufferTimePeriod": "5m",
        "invalidRecordsThresholdPercent": "30",
        "invalidRecordsThresholdCount": "100000",
        "tableMaxNumTasks": "40",
        "validDocIdsType": "SNAPSHOT"
      },
      "SegmentGenerationAndPushTask": {
        "schedule": "0 0 * * * ?"
      }
    }
  },
  "upsertConfig": {
    "mode": "FULL",
    "metadataTTL": 0,
    "deleteRecordColumn": "delete_record",
    "dropOutOfOrderRecord": true,
    "enableSnapshot": true,
    "deletedKeysTTL": 0,
    "enablePreload": false,
    "consistencyMode": "SNAPSHOT",
    "upsertViewRefreshIntervalMs": 3000,
    "hashFunction": "NONE",
    "defaultPartialUpsertStrategy": "OVERWRITE",
    "allowPartialUpsertConsumptionDuringCommit": false
  }
}
x
how do you generate the segment name ? it should be in the format of
Copy code
tableName__partitionGroupId__sequenceNumber__creationTime
p
The s3 location contains files with data json strings in every new line. Do I need to generate the segments? The task worked for offline table.
Or is there any config to generate the segments name?
x
yes, you need to repartition data based on the partition then generate segments with name follow the above pattern. Realtime segments has strict naming convention for the table requires
strictReplicaGroup
p
@Xiang Fu I am unable to generate the segment name in the required format
tableName__partitionGroupId__sequenceNumber__creationTime
Any pointers here? tried using:
Copy code
segmentNameGeneratorSpec:
  type: normalizedDate
  configs:
    timeColumnName: created_at
    timeFormat: '1:MILLISECONDS:EPOCH'
    segment.name.prefix: 'captain_kpi_offers_REALTIME'
    exclude.sequence.id: false
x
There is an
UploadedRealtimeSegmentNameGenerator
p
Tried this as well but it doesn’t generate segment name in this format
x
ok, how do you configure the batch job? I think you might need to follow the code to figure out this logic basically the job should use
UploadedRealtimeSegmentNameGenerator
to create a segment name.
p
Sure. Let me check. Thanks