Kiril Kalchev
07/18/2025, 9:10 PM"auctionsStats__6__13__20250703T1233Z": {
"Server_pinot-prod-server-0.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "OFFLINE",
"Server_pinot-prod-server-1.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "OFFLINE",
"Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098": "OFFLINE"
},
When I try to download the segments again, I get an error saying they are not in my deepstore. However, queries seem to work normally.
Is it expected for segments to be reported as offline and missing in deepstore? What exactly does offline mean as a segment status?
Bellow are the latest messages for the above segment:
INFO 2025-07-18T05:35:05.820609035Z [resource.labels.containerName: server] 2025/07/18 05:35:05.820 INFO [HttpClient] [auctionsStats__6__13__20250703T1233Z] Sending request: http://pinot-prod-controller-1.pinot-prod-controller-headless.pinot.svc.cluster.local:9000/segmentStoppedConsuming?reason=org.apache.pinot.shaded.org.apache.kafka.common.KafkaException&streamPartitionMsgOffset=0&instance=Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=auctionsStats__6__13__20250703T1233Z to controller: pinot-prod-controller-1.pinot-prod-controller-headless.pinot.svc.cluster.local, version: Unknown
INFO 2025-07-18T05:35:05.821542868Z [resource.labels.containerName: server] 2025/07/18 05:35:05.821 INFO [ServerSegmentCompletionProtocolHandler] [auctionsStats__6__13__20250703T1233Z] Controller response {"status":"PROCESSED","streamPartitionMsgOffset":null,"isSplitCommitType":true,"buildTimeSec":-1} for http://pinot-prod-controller-1.pinot-prod-controller-headless.pinot.svc.cluster.local:9000/segmentStoppedConsuming?reason=org.apache.pinot.shaded.org.apache.kafka.common.KafkaException&streamPartitionMsgOffset=0&instance=Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=auctionsStats__6__13__20250703T1233Z
INFO 2025-07-18T05:35:05.821571462Z [resource.labels.containerName: server] 2025/07/18 05:35:05.821 INFO [RealtimeSegmentDataManager_auctionsStats__6__13__20250703T1233Z] [auctionsStats__6__13__20250703T1233Z] Got response {"status":"PROCESSED","streamPartitionMsgOffset":null,"isSplitCommitType":true,"buildTimeSec":-1}
INFO 2025-07-18T05:35:05.983729827Z [resource.labels.containerName: server] 2025/07/18 05:35:05.976 INFO [local_8098 - SegmentOnlineOfflineStateModel] [HelixTaskExecutor-message_handle_thread_7] SegmentOnlineOfflineStateModel.onBecomeOfflineFromConsuming() : ZnRecord=cc787368-9a93-42f3-8588-ebefe88f2a07, {CREATE_TIMESTAMP=1752816905933, ClusterEventName=IdealStateChange, EXECUTE_START_TIMESTAMP=1752816905976, EXE_SESSION_ID=300627ec087008e, FROM_STATE=CONSUMING, MSG_ID=cc787368-9a93-42f3-8588-ebefe88f2a07, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=auctionsStats__6__13__20250703T1233Z, READ_TIMESTAMP=1752816905959, RESOURCE_NAME=auctionsStats_REALTIME, RESOURCE_TAG=auctionsStats_REALTIME, RETRY_COUNT=3, SRC_NAME=pinot-prod-controller-2.pinot-prod-controller-headless.pinot.svc.cluster.local_9000, SRC_SESSION_ID=2006281fc800087, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_pinot-prod-server-2.pinot-prod-server-headless.pinot.svc.cluster.local_8098, TGT_SESSION_ID=300627ec087008e, TO_STATE=OFFLINE}{}{}, Stat=Stat {_version=0, _creationTime=1752816905946, _modifiedTime=1752816905946, _ephemeralOwner=0}
INFO 2025-07-18T05:35:05.984995178Z [resource.labels.containerName: server] 2025/07/18 05:35:05.983 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_7] Removing segment: auctionsStats__6__13__20250703T1233Z from table: auctionsStats_REALTIME
INFO 2025-07-18T05:35:05.985038958Z [resource.labels.containerName: server] 2025/07/18 05:35:05.983 INFO [auctionsStats_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread_7] Removing segment: auctionsStats__6__13__20250703T1233Z from table: auctionsStats_REALTIME
INFO 2025-07-18T05:35:05.985045952Z [resource.labels.containerName: server] 2025/07/18 05:35:05.983 INFO [auctionsStats_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread_7] Closing segment: auctionsStats__6__13__20250703T1233Z of table: auctionsStats_REALTIME
INFO 2025-07-18T05:35:05.985110098Z [resource.labels.containerName: server] 2025/07/18 05:35:05.984 INFO [MutableSegmentImpl_auctionsStats__6__13__20250703T1233Z_auctionsStats] [HelixTaskExecutor-message_handle_thread_7] Trying to close RealtimeSegmentImpl : auctionsStats__6__13__20250703T1233Z
INFO 2025-07-18T05:35:05.985117081Z [resource.labels.containerName: server] 2025/07/18 05:35:05.984 INFO [auctionsStats_REALTIME-6-ConcurrentMapPartitionUpsertMetadataManager] [HelixTaskExecutor-message_handle_thread_7] Skip removing untracked (replaced or empty) segment: auctionsStats__6__13__20250703T1233Z
INFO 2025-07-18T05:35:05.987557288Z [resource.labels.containerName: server] 2025/07/18 05:35:05.987 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_7] Deleted file /var/pinot/server/data/index/auctionsStats_REALTIME/consumers/auctionsStats__6__13__20250703T1233Z.0
INFO 2025-07-18T05:35:05.990545309Z [resource.labels.containerName: server] 2025/07/18 05:35:05.990 INFO [auctionsStats_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread_7] Closed segment: auctionsStats__6__13__20250703T1233Z of table: auctionsStats_REALTIME
INFO 2025-07-18T05:35:05.990570191Z [resource.labels.containerName: server] 2025/07/18 05:35:05.990 INFO [auctionsStats_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread_7] Removed segment: auctionsStats__6__13__20250703T1233Z from table: auctionsStats_REALTIME
INFO 2025-07-18T05:35:05.990578459Z [resource.labels.containerName: server] 2025/07/18 05:35:05.990 INFO [HelixInstanceDataManager] [HelixTaskExecutor-message_handle_thread_7] Removed segment: auctionsStats__6__13__20250703T1233Z from table: auctionsStats_REALTIME
INFO 2025-07-18T06:15:57.880369560Z [resource.labels.containerName: controller] 2025/07/18 06:15:57.880 INFO [PinotLLCRealtimeSegmentManager] [pool-10-thread-7] Repairing segment: auctionsStats__6__13__20250703T1233Z which is OFFLINE for all instances in IdealStateMayank
Kiril Kalchev
07/18/2025, 9:58 PMKiril Kalchev
07/18/2025, 10:00 PMMayank
Kiril Kalchev
07/19/2025, 7:24 AMKiril Kalchev
07/19/2025, 7:34 AMKiril Kalchev
07/19/2025, 7:45 AMMayank
Kiril Kalchev
07/19/2025, 1:24 PMKiril Kalchev
07/19/2025, 1:27 PMKiril Kalchev
07/19/2025, 1:28 PMKiril Kalchev
07/19/2025, 1:29 PMMayank
Mayank
Kiril Kalchev
07/19/2025, 5:27 PMMayank
Kiril Kalchev
07/23/2025, 2:48 PMMayank