Vipin Rohilla
05/19/2025, 1:58 PM/opt/pinot/conf# grep -ae "testdelta1__0__0__20250519T1330Z" /var/loghdp/pinot/pinot-server.log
2025/05/19 19:00:10.933 INFO [nb6_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_0] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, {CREATE_TIMESTAMP=1747661410779, ClusterEventName=IdealStateChange, EXECUTE_START_TIMESTAMP=1747661410908, EXE_SESSION_ID=20886635e6e0c43, FROM_STATE=OFFLINE, MSG_ID=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=testdelta1__0__0__20250519T1330Z, READ_TIMESTAMP=1747661410863, RESOURCE_NAME=testdelta1_REALTIME, RESOURCE_TAG=testdelta1_REALTIME, RETRY_COUNT=3, SRC_NAME=[CONTROLLER], SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=[SERVER], TGT_SESSION_ID=20886635e6e0c43, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747661410784, _modifiedTime=1747661410784, _ephemeralOwner=0}
Any insights on why this is happening and how to fix it? We have verified Kafka connectivity and data format.Xiang Fu
Vipin Rohilla
05/20/2025, 9:49 AM2025/05/19 17:07:39.036 INFO [ServerQueryExecutorV1Impl] [Start a Pinot [SERVER]] Query executor started
2025/05/19 17:07:39.038 INFO [ServerInstance] [Start a Pinot [SERVER]] Finish starting query server
2025/05/19 17:07:39.055 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Pinot server ready
2025/05/19 17:07:39.062 INFO [ServerQueriesDisabledTracker] [Start a Pinot [SERVER]] Tracking server queries disabled.
2025/05/19 17:07:39.068 INFO [PinotServiceManager] [Start a Pinot [SERVER]] Pinot Server instance [Server_stg-hdppinotserver103_8098] is Started...
2025/05/19 17:07:39.068 INFO [StartServiceManagerCommand] [Start a Pinot [SERVER]] Started Pinot [SERVER] instance [Server_stg-hdppinotserver103_8098] at 56.684s since launch
2025/05/19 19:00:10.933 INFO [xxxxxx_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_0] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, {CREATE_TIMESTAMP=1747661410779, ClusterEventName=IdealStateChange, EXECUTE_START_TIMESTAMP=1747661410908, EXE_SESSION_ID=20886635e6e0c43, FROM_STATE=OFFLINE, MSG_ID=77f7e43b-0cfc-4a94-aa32-25e9c2121de6, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=testdelta1__0__0__20250519T1330Z, READ_TIMESTAMP=1747661410863, RESOURCE_NAME=testdelta1_REALTIME, RESOURCE_TAG=testdelta1_REALTIME, RETRY_COUNT=3, SRC_NAME=stg-hdppinotcontroller102_9000, SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_stg-hdppinotserver103_8098, TGT_SESSION_ID=20886635e6e0c43, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747661410784, _modifiedTime=1747661410784, _ephemeralOwner=0}
2025/05/19 19:00:10.944 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_0] Creating table data manager for table: testdelta1_REALTIME
2025/05/19 19:00:11.105 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_0] Initializing table data manager for table: testdelta1_REALTIMEXiang Fu
Vipin Rohilla
05/20/2025, 1:34 PM2025/05/20 18:55:03.413 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Starting server admin application on: http://0.0.0.0:8097
2025/05/20 18:55:03.505 INFO [xxxxxx_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_0] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=dcc53a88-78b5-42f4-9155-dd2ca5f18a86, {CREATE_TIMESTAMP=1747747503394, ClusterEventName=CurrentStateChange, EXECUTE_START_TIMESTAMP=1747747503462, EXE_SESSION_ID=20886635e6e0c95, FROM_STATE=OFFLINE, MSG_ID=dcc53a88-78b5-42f4-9155-dd2ca5f18a86, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=test__3__0__20250520T1322Z, READ_TIMESTAMP=1747747503411, RESOURCE_NAME=test_REALTIME, RESOURCE_TAG=test_REALTIME, RETRY_COUNT=3, SRC_NAME=xxxxxxpinotcontroller102.xxxxxx_9000, SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_xxxxxxpinotserver102.xxxxxx_8098, TGT_SESSION_ID=20886635e6e0c95, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747747503396, _modifiedTime=1747747503396, _ephemeralOwner=0}
2025/05/20 18:55:03.501 INFO [xxxxxx_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_1] SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() : ZnRecord=f18fdfa3-c3ec-4a8e-b2e4-e940a1bb268e, {CREATE_TIMESTAMP=1747747503394, ClusterEventName=CurrentStateChange, EXECUTE_START_TIMESTAMP=1747747503466, EXE_SESSION_ID=20886635e6e0c95, FROM_STATE=OFFLINE, MSG_ID=f18fdfa3-c3ec-4a8e-b2e4-e940a1bb268e, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=test__0__0__20250520T1322Z, READ_TIMESTAMP=1747747503417, RESOURCE_NAME=test_REALTIME, RESOURCE_TAG=test_REALTIME, RETRY_COUNT=3, SRC_NAME=xxxxxxpinotcontroller102.xxxxxx_9000, SRC_SESSION_ID=108861e1f2d0c39, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_xxxxxxpinotserver102.xxxxxx_8098, TGT_SESSION_ID=20886635e6e0c95, TO_STATE=CONSUMING}{}{}, Stat=Stat {_version=0, _creationTime=1747747503396, _modifiedTime=1747747503396, _ephemeralOwner=0}
2025/05/20 18:55:03.515 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_0] Creating table data manager for table: test_REALTIME
2025/05/20 18:55:03.812 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_0] Initializing table data manager for table: test_REALTIME
2025/05/20 18:55:03.819 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_1] Creating table data manager for table: test_REALTIME
2025/05/20 18:55:03.839 INFO [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread_1] Initializing table data manager for table: test_REALTIME
ms to scan 1 urls, producing 20354 keys and 120613 values
2025/05/20 18:55:18.052 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Initializing QueryRewriterFactory
2025/05/20 18:55:18.094 INFO [ServiceStatus] [Start a Pinot [SERVER]] Monitoring 1 resources: [test_REALTIME] for start up of instance Server_xxxxxxpinotserver102.xxxxxx_8098
2025/05/20 18:55:18.094 INFO [ServiceStatus] [Start a Pinot [SERVER]] Monitoring 1 resources: [test_REALTIME] for start up of instance Server_xxxxxxpinotserver102.xxxxxx_8098
2025/05/20 18:55:18.095 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Registering service status handler
2025/05/20 18:55:18.095 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Starting startup service status check
2025/05/20 18:55:18.098 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__0__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.099 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__3__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.101 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__0__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.102 ERROR [ServiceStatus] [Start a Pinot [SERVER]] Resource: test_REALTIME, partition: test__3__0__20250520T1322Z is in ERROR state
2025/05/20 18:55:18.102 INFO [BaseServerStarter] [Start a Pinot [SERVER]] Service status is GOOD after 7ms
2025/05/20 18:55:18.107 WARN [RealtimeConsumptionRateManager] [Start a Pinot [SERVER]] Invalid server consumption rate limit: 0.0, throttling is disabled
2025/05/20 18:55:18.107 INFO [ServerInstance] [Start a Pinot [SERVER]] Starting query server
2025/05/20 18:55:18.107 INFO [ServerInstance] [Start a Pinot [SERVER]] Starting query executorXiang Fu
Xiang Fu
Vipin Rohilla
05/20/2025, 3:29 PM{
"tableName": "test",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "ts",
"schemaName": "test",
"replicasPerPartition": "4"
},
"tenants": {},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.topic.name": "third_topic",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "stg-xxxxx101.xxxxx:6667,stg-xxxxx102.xxxxx:6667,stg-xxxxx103.xxxxx:6667",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.threshold.segment.size": "50M",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
},
"metadata": {
"customConfigs": {}
}
}Xiang Fu
"replicasPerPartition": "4"Xiang Fu
Xiang Fu
Xiang Fu
Vipin Rohilla
05/21/2025, 7:27 AMNihal Jain
05/22/2025, 5:57 PMNihal Jain
05/22/2025, 7:59 PMVipin Rohilla
05/23/2025, 2:30 AMVipin Rohilla
05/23/2025, 2:36 AMXiang Fu