When a segment file fails to be uploaded to the deep store (here: https://github.com/apache/pinot/blob/30c4635bfeee88f88aa9c9f63b93bcd4a650607f/pino[…]he/pinot/core/data/manager/realtime/PinotFSSegmentUploader.java), the tar.gz file gets removed. Does this segment ever get retried? I do have

controller.realtime.segment.deepStoreUploadRetryEnabled=true

but have not seen it happen. This is a controller setting, but it’s the server doing the upload.

The retry is/should-be on the build and push side that is uploading, iirc.

on the build and push side

what does that mean?

Sorry, should have not assumed, is this realtime or offline?

realtime

Ok, my comment assumed offline. Looking at the code, I don’t see a retry. Do you know why the push failed in your case?

something timed out, but anything can happen with external systems. For what situations is

controller.realtime.segment.deepStoreUploadRetryEnabled

being used?

Ok, looking at the code, the retry is not immediate. But the retry happens with the periodic segment validation task.

Ah thanks I’ll try that. How often does this run automatically?

It is configurable, and defaults to 1hr.

I looked at http://pinot-controller:9000/help#/Task/listTaskTypes but it returns an empty list. How do I find out what task I can trigger manually?

There is also an api in Swagger that returns those:

[
  "RetentionManager",
  "OfflineSegmentIntervalChecker",
  "RealtimeSegmentValidationManager",
  "BrokerResourceValidationManager",
  "SegmentStatusChecker",
  "SegmentRelocator",
  "MinionInstancesCleanupTask",
  "TaskMetricsEmitter"
]

I tried

Task/listTaskTypes

but that returns an empty list

/periodictask/names

in Swagger

ooh! Many thanks!