Hi Team, pinot tables are stuck at updating status. While checking Ideal state and External View for all these tables(through pinot UI), server-4 is making the issue. But I could not see any error messages in server-4 logs. Some tables, replication is 1 and those tables are failed in query as segment in server-4 are missing. Could you please help me to identify the root cause for this issue ?
x
Xiang Fu
05/06/2025, 10:19 AM
have you tried to restart server-4 and capture any logs ?
Xiang Fu
05/06/2025, 10:19 AM
any cpu/memory/disk issue could be identified?
p
Prijo Pauly
05/06/2025, 10:39 AM
yes, I restarted, server-4 everything works fine now. Captured the logs before restarting it, but couldn't find any reason for failure. cpu/memory/disk everything was ok before restarting.
x
Xiang Fu
05/06/2025, 12:07 PM
i see, we need to check why segment got into bad status
Xiang Fu
05/06/2025, 12:07 PM
you need to check why the previous segment got failed
p
Prasad V
05/08/2025, 4:12 AM
Is it a right option to restart server ?
x
Xiang Fu
05/08/2025, 5:17 AM
If you found a particular server is bad, then you can restart it.
Xiang Fu
05/08/2025, 5:17 AM
but I bet it will be some error messages
p
Prijo Pauly
05/28/2025, 4:39 AM
@Xiang Fu It happened again, this time for server 3, no hint from server logs. But getting below logs in controller nodes
"Postponing deletion of 1 segments from table t1", where t1 has the state 'Updating'
Prijo Pauly
05/28/2025, 4:42 AM
2025/05/28 044159.233 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1
2025/05/28 044202.163 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table t1:[]
2025/05/28 044202.163 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1
2025/05/28 044205.275 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table t1:[]
2025/05/28 044205.275 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1
2025/05/28 044215.139 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table t1:[]
2025/05/28 044215.139 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1
Prijo Pauly
05/28/2025, 4:48 AM
hi @Jayesh Asrani, expecting your help on above issue