QQ: Is there a certain task we can run to force a server to re-upload it's segment to the deep store (in our case S3)? We have a situation where a realtime server failed to upload to S3, and then the segment was offloaded to offline servers, The offline servers were able to fetch the segment from their online peers to successfully load the segment, but the segment is still in a weird state where it is missing from the deep-store/S3. Should some periodic task be running to check on this, or can we run some manual controller task to "heal" the situation?

This can only happen if you use peer download. Are you using that? That feature is for a non-HA deepstore (eg HDFS is some deployments)

Yes, in our case, the servers are sometimes falling back to peer download because of an aws credential issue on our end when uploading to S3. But in theory, if there is any timeout or other issue to uploading to S3, and retry attempts are exceeded, this case would be hit

May be you can create a GH issue on this? Seems like a fair ask to me.

Sure I can do that