Hi Team We are about to go live with pinot setup . We are currently on pinot version 1.4 and we have deployed the cluster on AKS via helm chart. Now every time we do some changes like increasing the pod memory or other upgrades it it restarts the pod which is expected but the problem is few tables(every time something different) never come back with all the segments as good and they are bad . We have configured ADLS as the deep store to avoid segment corruption while upgrades but yet it did not help us and we noticed errors :

[HelixTaskExecutor-message_handle_thread_34] Caught exception while processing
SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline() for table: gas_pos_REALTIME,      
segment: gas_pos__2__0__20260219T0906Z
> org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3       
attempts
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) 
        at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegme
ntFetcher.java:74)
        at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)    
        at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed    
after 3 attempts
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) 
        at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegme
ntFetcher.java:74)
  2026/02/25 11:35:54.580 ERROR [HelixStateTransitionHandler] 
[HelixTaskExecutor-message_handle_thread_34] Skip internal error. errCode: ERROR, errMsg:    
null
  2026/02/25 11:35:56.252 ERROR [local_8098 - SegmentOnlineOfflineStateModel]
[HelixTaskExecutor-message_handle_thread_33] Caught exception while processing
SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline() for table: gas_pos_REALTIME,      
segment: gas_pos__4__0__20260219T0906Z
> org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3       
attempts
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) 
        at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegme
ntFetcher.java:74)
        at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)    
        at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed    
after 3 attempts
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) 
        at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegme
ntFetcher.java:74)
  2026/02/25 11:35:56.258 ERROR [HelixTask] [HelixTaskExecutor-message_handle_thread_33]     
Message execution failed. msgId: 07c9d7cd-5771-46eb-b7b1-8135f0a6033c, errorMsg:
java.lang.reflect.InvocationTargetException
  2026/02/25 11:35:56.261 ERROR [HelixStateTransitionHandler]
[HelixTaskExecutor-message_handle_thread_33] Skip internal error. errCode: ERROR, errMsg:    
null
> 2026/02/25 11:35:56.687 ERROR [local_8098 - SegmentOnlineOfflineStateModel]
[HelixTaskExecutor-message_handle_thread_9] Caught exception while processing
SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline() for table: gas_pos_REALTIME,      
segment: gas_pos__0__0__20260219T0906Z
> org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 
attempts
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) 
        at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegme
ntFetcher.java:74)
        at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)    
        at java.base/java.lang.Thread.run(Thread.java:840)
> 2026/02/25 11:35:56.687 ERROR [HelixStateTransitionHandler]
[HelixTaskExecutor-message_handle_thread_9] Exception while executing a state transition     
task gas_pos__0__0__20260219T0906Z
  java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)    
        at java.base/java.lang.Thread.run(Thread.java:840)
> Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed    
after 3 attempts
        at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) 
        at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegme
ntFetcher.java:74)

So My question is : 1. How do we handle the upgrades of the pods without making the segment as bad 2. Is there a way to gracefully shutdown the system apply the Rolling upgrade and than restart the system I tried to Reload the segments . also reset the segments but the bad segments never came back

If you do rolling restart of components in the order of Controller -> Broker -> Server you should not run into the issue. Are you seeing any exceptions/errors? What you posted above is the outer exception that was caught.

this is the exception we in the server logs . I think we are not following the order which can be the problem