Hi All, I am facing an issue where the table test ...
# troubleshooting
v
Hi All, I am facing an issue where the table test goes into a bad state immediately after creation. We can produce and consume data in the Kafka topic, but the table’s segment testdelta1__0__0__20250519T1330Z fails to transition to CONSUMING and moves to ERROR. The controller reports the table as UNHEALTHY with the error: "Did not get any response from servers for segment: testdelta1__0__0__20250519T1330Z". Server logs:
Copy code
/opt/pinot/conf# grep -ae "testdelta1__0__0__20250519T1330Z" /var/loghdp/pinot/pinot-server.log
2025/05/19 19:00:10.933 INFO [nb6_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_0] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, {CREATE_TIMESTAMP=1747661410779, ClusterEventName=IdealStateChange, EXECUTE_START_TIMESTAMP=1747661410908, EXE_SESSION_ID=20886635e6e0c43, FROM_STATE=OFFLINE, MSG_ID=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=testdelta1__0__0__20250519T1330Z, READ_TIMESTAMP=1747661410863, RESOURCE_NAME=testdelta1_REALTIME, RESOURCE_TAG=testdelta1_REALTIME, RETRY_COUNT=3, SRC_NAME=[CONTROLLER], SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=[SERVER], TGT_SESSION_ID=20886635e6e0c43, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747661410784, _modifiedTime=1747661410784, _ephemeralOwner=0}
Any insights on why this is happening and how to fix it? We have verified Kafka connectivity and data format.
x
Can you check pinot server for any error logs during segment creation
v
cant find any ERROR logs during the segment creation, below are the logs i see during segment creation
Copy code
2025/05/19 17:07:39.036 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Query executor started
2025/05/19 17:07:39.038 INFO [ServerInstance] [Start a Pinot [SERVER]] Finish starting query server
2025/05/19 17:07:39.055 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Pinot server ready
2025/05/19 17:07:39.062 INFO [ServerQueriesDisabledTracker] [Start a Pinot [SERVER]] Tracking server queries disabled.
2025/05/19 17:07:39.068 INFO [PinotServiceManager] [Start a Pinot [SERVER]] Pinot Server instance [Server_stg-hdppinotserver103_8098] is Started...
2025/05/19 17:07:39.068 INFO [StartServiceManagerCommand] [Start a Pinot [SERVER]] Started Pinot [SERVER] instance [Server_stg-hdppinotserver103_8098] at 56.684s since launch
2025/05/19 19:00:10.933 INFO [xxxxxx_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_0] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, {CREATE_TIMESTAMP=1747661410779, ClusterEventName=IdealStateChange, EXECUTE_START_TIMESTAMP=1747661410908, EXE_SESSION_ID=20886635e6e0c43, FROM_STATE=OFFLINE, MSG_ID=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=testdelta1__0__0__20250519T1330Z, READ_TIMESTAMP=1747661410863, RESOURCE_NAME=testdelta1_REALTIME, RESOURCE_TAG=testdelta1_REALTIME, RETRY_COUNT=3, SRC_NAME=stg-hdppinotcontroller102_9000, SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_stg-hdppinotserver103_8098, TGT_SESSION_ID=20886635e6e0c43, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747661410784, _modifiedTime=1747661410784, _ephemeralOwner=0}
2025/05/19 19:00:10.944 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_0] Creating table data manager for table: testdelta1_REALTIME
2025/05/19 19:00:11.105 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_0] Initializing table data manager for table: testdelta1_REALTIME
x
can you enter into the docker/k8s container to check the server log file ?
v
yes, We are using VM. i have the log file with me.
Copy code
2025/05/20 18:55:03.413 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Starting server admin application on: http://0.0.0.0:8097
2025/05/20 18:55:03.505 INFO [xxxxxx_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_0] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=dcc53a88-78b5-42f4-9155-dd2ca5f18a86, {CREATE_TIMESTAMP=1747747503394, ClusterEventName=CurrentStateChange, EXECUTE_START_TIMESTAMP=1747747503462, EXE_SESSION_ID=20886635e6e0c95, FROM_STATE=OFFLINE, MSG_ID=dcc53a88-78b5-42f4-9155-dd2ca5f18a86, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=test__3__0__20250520T1322Z, READ_TIMESTAMP=1747747503411, RESOURCE_NAME=test_REALTIME, RESOURCE_TAG=test_REALTIME, RETRY_COUNT=3, SRC_NAME=xxxxxxpinotcontroller102.xxxxxx_9000, SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_xxxxxxpinotserver102.xxxxxx_8098, TGT_SESSION_ID=20886635e6e0c95, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747747503396, _modifiedTime=1747747503396, _ephemeralOwner=0}
2025/05/20 18:55:03.501 INFO [xxxxxx_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_1] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=f18fdfa3-c3ec-4a8e-b2e4-e940a1bb268e, {CREATE_TIMESTAMP=1747747503394, ClusterEventName=CurrentStateChange, EXECUTE_START_TIMESTAMP=1747747503466, EXE_SESSION_ID=20886635e6e0c95, FROM_STATE=OFFLINE, MSG_ID=f18fdfa3-c3ec-4a8e-b2e4-e940a1bb268e, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=test__0__0__20250520T1322Z, READ_TIMESTAMP=1747747503417, RESOURCE_NAME=test_REALTIME, RESOURCE_TAG=test_REALTIME, RETRY_COUNT=3, SRC_NAME=xxxxxxpinotcontroller102.xxxxxx_9000, SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_xxxxxxpinotserver102.xxxxxx_8098, TGT_SESSION_ID=20886635e6e0c95, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747747503396, _modifiedTime=1747747503396, _ephemeralOwner=0}
2025/05/20 18:55:03.515 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_0] Creating table data manager for table: test_REALTIME
2025/05/20 18:55:03.812 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_0] Initializing table data manager for table: test_REALTIME
2025/05/20 18:55:03.819 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_1] Creating table data manager for table: test_REALTIME
2025/05/20 18:55:03.839 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_1] Initializing table data manager for table: test_REALTIME

 ms to scan 1 urls, producing 20354 keys and 120613 values
2025/05/20 18:55:18.052 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Initializing QueryRewriterFactory
2025/05/20 18:55:18.094 INFO [ServiceStatus] [Start a Pinot [SERVER]] Monitoring 1 resources: [test_REALTIME] for start up of instance Server_xxxxxxpinotserver102.xxxxxx_8098
2025/05/20 18:55:18.094 INFO [ServiceStatus] [Start a Pinot [SERVER]] Monitoring 1 resources: [test_REALTIME] for start up of instance Server_xxxxxxpinotserver102.xxxxxx_8098
2025/05/20 18:55:18.095 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Registering service status handler
2025/05/20 18:55:18.095 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Starting startup service status check
2025/05/20 18:55:18.098 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__0__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.099 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__3__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.101 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__0__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.102 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__3__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.102 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Service status is GOOD after 7ms
2025/05/20 18:55:18.107 WARN [RealtimeConsumptionRateManager] [Start a Pinot [SERVER]] Invalid server consumption rate limit: 0.0, throttling is disabled
2025/05/20 18:55:18.107 INFO [ServerInstance] [Start a Pinot [SERVER]] Starting query server
2025/05/20 18:55:18.107 INFO [ServerInstance] [Start a Pinot [SERVER]] Starting query executor
x
yeah, this is too less, no stacktrace ?
if so can you share the full table config?
v
yeah there's no stack trace
Copy code
{
  "tableName": "test",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "ts",
    "schemaName": "test",
    "replicasPerPartition": "4"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "third_topic",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "stg-xxxxx101.xxxxx:6667,stg-xxxxx102.xxxxx:6667,stg-xxxxx103.xxxxx:6667",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.threshold.segment.size": "50M",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
x
so you have 4 pinot servers ?
Copy code
"replicasPerPartition": "4"
this is weird though
the configs seems to be ok
can you drop the entire log file from the container ?
v
My bad, its permission issue on server data dir. fixed it. Thanks for the help though @Xiang Fu appreciate it
n
Hi @vipin can you share details on what was the file structure and permission that caused this issue. I want to reproduce this issue locally for a fix
@Vipin Rohilla I have raised a fix at https://github.com/apache/pinot/pull/15876, I hope this covers your failure scenario. Please, let me know if otherwise.
v
Basically /data/pinotserver was owned by root and permission were 600. I have set the ownership to pinot:pinot and gave 644, it worked
Thanks Nihal, yes it covers the scenario
x
thanks!!!