Hi Team, pinot tables are stuck at updating status...
# troubleshooting
p
Hi Team, pinot tables are stuck at updating status. While checking Ideal state and External View for all these tables(through pinot UI), server-4 is making the issue. But I could not see any error messages in server-4 logs. Some tables, replication is 1 and those tables are failed in query as segment in server-4 are missing. Could you please help me to identify the root cause for this issue ?
x
have you tried to restart server-4 and capture any logs ?
any cpu/memory/disk issue could be identified?
p
yes, I restarted, server-4 everything works fine now. Captured the logs before restarting it, but couldn't find any reason for failure. cpu/memory/disk everything was ok before restarting.
x
i see, we need to check why segment got into bad status
you need to check why the previous segment got failed
p
Is it a right option to restart server ?
x
If you found a particular server is bad, then you can restart it.
but I bet it will be some error messages
p
@Xiang Fu It happened again, this time for server 3, no hint from server logs. But getting below logs in controller nodes "Postponing deletion of 1 segments from table t1", where t1 has the state 'Updating'
2025/05/28 044159.233 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1 2025/05/28 044202.163 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table t1:[] 2025/05/28 044202.163 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1 2025/05/28 044205.275 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table t1:[] 2025/05/28 044205.275 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1 2025/05/28 044215.139 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Deleted 0 segments from table t1:[] 2025/05/28 044215.139 INFO [SegmentDeletionManager] [PinotHelixResourceManagerExecutorService] Postponing deletion of 1 segments from table t1
hi @Jayesh Asrani, expecting your help on above issue