Hi team, I got an error in the segments while usin...
# troubleshooting
s
Hi team, I got an error in the segments while using pinot upsert. So I have 2 servers configured, but both of them were error, the status becomes bad when it creates into a new segment, but the thing is, the previous created segments turns into bad and the new created segments were okay, but I can do some query in it when I delete the previous segments.
Copy code
"segmentName": "20250313T0831Z",
  "serverState": {
    "Server_1": {
      "idealState": "ONLINE",
      "externalView": "ERROR",
      "segmentSize": null,
      "consumerInfo": null,
      "errorInfo": null
    }
Here's my schema:
Copy code
{
  "schemaName": "cpats_stocks",
  "primaryKeyColumns": [
    "c_project_id"
  ],
  "dimensionFieldSpecs": [
    {
      "name": "c_project_id",
      "dataType": "LONG",
      "notNull": false
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "z_qty_total",
      "dataType": "LONG",
      "notNull": false
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "__metadata$recordTimestamp",
      "dataType": "TIMESTAMP",
      "format": "SIMPLE_DATE_FORMAT|yyyy-MM-dd",
      "granularity": "1:DAYS"
    }
  ]
}
Here's my table:
Copy code
{
  "tableName": "cpats_stocks",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "schemaName": "cpats_stocks",
    "timeColumnName": "__metadata$recordTimestamp",
    "replicasPerPartition": "2",
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "1",
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "minimizeDataMovement": false
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant"
  },
  "upsertConfig": {
    "mode": "FULL",
    "metadataTTL": 86400,
    "dropOutOfOrderRecord": false,
    "enableSnapshot": true,
    "deletedKeysTTL": 0,
    "defaultPartialUpsertStrategy": "OVERWRITE",
    "enablePreload": true,
    "consistencyMode": "SNAPSHOT",
    "upsertViewRefreshIntervalMs": 3000,
    "allowPartialUpsertConsumptionDuringCommit": false,
    "hashFunction": "MD5"
  },
  "instanceAssignmentConfigMap": {
    "CONSUMING": {
      "tagPoolConfig": {
        "tag": "DefaultTenant_REALTIME"
      },
      "replicaGroupPartitionConfig": {
        "replicaGroupBased": true,
        "numReplicaGroups": 1,
        "partitionColumn": "c_project_id",
        "numPartitions": 2,
        "numInstancesPerPartition": 1
      }
    }
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "tableIndexConfig": {
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "cpats_stocks",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "broker:9092",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.metadata.populate": true,
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    },
    "segmentPartitionConfig": {
      "columnPartitionMap": {
        "c_project_id": {
          "functionName": "Hashcode",
          "numPartitions": 2
        }
      }
    },
    "invertedIndexColumns": [
      "c_project_id"
    ],
    "aggregateMetrics": false,
    "enableDefaultStarTree": false,
    "nullHandlingEnabled": true,
    "rangeIndexVersion": 2,
    "optimizeDictionaryForMetrics": false,
    "optimizeDictionary": false,
    "loadMode": "MMAP",
    "enableDynamicStarTreeCreation": false,
    "columnMajorSegmentBuilderEnabled": true,
    "noDictionarySizeRatioThreshold": 0.85,
    "autoGeneratedInvertedIndex": false,
    "createInvertedIndexDuringSegmentGeneration": false
  },
  "metadata": {
    "customConfigs": {}
  },
  "ingestionConfig": {
    "continueOnError": false,
    "rowTimeValueCheck": false,
    "segmentTimeValueCheck": true
  },
  "isDimTable": false
}
Copy code
2025/03/13 07:55:40.817 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(d8c94f54_DEFAULT)] Event d8c94f54_DEFAULT : Unable to find a next state for resource: cpats_stocks_REALTIME partition: cpats_stocks__0__0__20250313T0755Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
I got this error
m
Check table debug api, it should bubble up any exceptions related to segment commit
s
Copy code
[
  {
    "tableName": "cpats_stocks_REALTIME",
    "numSegments": 4,
    "numServers": 2,
    "numBrokers": 2,
    "segmentDebugInfos": [],
    "serverDebugInfos": [],
    "brokerDebugInfos": [],
    "tableSize": {
      "reportedSize": "-1 bytes",
      "estimatedSize": "-1 bytes"
    },
    "ingestionStatus": {
      "ingestionState": "UNHEALTHY",
      "errorMessage": "Did not get any response from servers for segment: cpats_stocks__0__3__20250314T0410Z"
    }
  }
]
This is what I get
this is my cluster configs:
Copy code
{
  "allowParticipantAutoJoin": "true",
  "enable.case.insensitive": "true",
  "pinot.broker.enable.query.limit.override": "false",
  "default.hyperloglog.log2m": "8"
}
m
Hmm, then check for segment name in the pinot controller and server log. It should show the error/exception
s
Pinot Controller log:
Copy code
2025/03/18 07:28:35.537 ERROR [CompletionServiceHelper] [grizzly-http-server-8] Connection error  in 'get segment size info from servers'. Details: org.apache.pinot.shaded.org.apache.http.conn.HttpHostConnectException: Connect to ip [/ip] failed: Connection refused

2025/03/19 02:31:51.828 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(1986b48b_DEFAULT)] Event 1986b48b_DEFAULT : Unable to find a next state for resource: cpats_stocks_REALTIME partition: cpats_stocks__0__481__20250319T0231Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:ONLINE
2025/03/19 02:31:51.897 ERROR [SegmentCompletionFSM_cpats_stocks__0__482__20250319T0231Z] [grizzly-http-server-14] Caught exception while committing segment metadata for segment: cpats_stocks__0__482__20250319T0231Z
java.lang.NullPointerException: Cannot invoke "java.util.Map.containsValue(Object)" because the return value of "org.apache.helix.model.IdealState.getInstanceStateMap(String)" is null
	at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:524) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:503) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.commitSegment(SegmentCompletionManager.java:1064) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager$SegmentCompletionFSM.segmentCommitEnd(SegmentCompletionManager.java:627) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentCommitEnd(SegmentCompletionManager.java:295) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommitEndWithMetadata(LLCSegmentCompletionHandlers.java:396) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at jdk.internal.reflect.GeneratedMethodAccessor307.invoke(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Pinot Server Log:
Copy code
ERROR [RealtimeSegmentDataManager_cpats_stock__0__0__20250305T0852Z] [cpats_stock__0__0__20250305T0852Z] Caught exception while transforming the record at offset: 0 , row: {
It started from offset 0 until the end of the whole topic is it because my setup?
@Mayank I use the default setup on the documentation
m
seems like the transform you specified doesn't work on the data
s
I was trying to convert the json data, so from kafka it was json file, then I want to convert it into a string and Long type of data in pinot. Since the data has some null values in the columns, I add
"nullHandlingEnabled": "true"
in the table index config.
in the table:
Copy code
...
"ingestionConfig": {
   "transformConfigs": [
         {
           "columnName": "c_project_id",
           "transformFunction": "JSONPATHDOUBLE(message, '$.c_project_id')"
         }
}
...
is it possible that the
zookeeper
have a conflict with the s`erver/host/broker`? Which make the
segments Error
@Mayank
m
No.
s
Does minion configuration effects it? Cause I didn't setup the minion
I just want to inform you that the problem was caused by pinot version, I use the 1.2.0 version, then I change the version to latest or 1.4.0 and the upsert works without failing on the segment
thanks for your help, really appreciate it
m
Interesting, thanks for sharing.