Has anyone faced the issue where the downloaded segments on the server are not consistent? https://github.com/apache/pinot/issues/9003

Temp disk issue?

this is recurring in our cluster

Is the tar.gz file on controller is valid?

two different server (for 2 replication) failed at the similar time. So, we suspect that the controller returns incorrect bytes and somehow it converges to the correct bytes later

So you’re storing segments on the controller? Or using some shared FS for deep storage?

its a shared fs

If you’re using a shared fs for deep storage, then the controller isn’t involved in moving segment data onto the servers. I would suspect an issue with your shared fs, or maybe networking problems (though that seems to always be the hand-waving reason for transient problems).

For shared FS, (I think it is NFS in this case), controller provides the http/https url for upload/download of the segments, so the payload has to go through controller.

Hmm, if there’s an NFS mount point on each of the Pinot servers, then wouldn’t they just directly read from NFS?

No, the NFS mount here is across controllers, not servers.

OK, but why wouldn’t you define NFS mount points on the servers? I assume having each server reading directly would be more efficient? Sorry, I don’t use NFS much (not in the past 5+ years, anyway) so I don’t have background in good/bad configuration choices.

NFS is not the right tech for the workload servers will run. Infact, someone tried recently in OSS, and it did not work for them. What you are saying already happens in case of deep-store like S3/GCP. In that case servers directly download from those deep-store, and bypass controller, since those deepstores provide a url for download. In case of NFS as deepstore, there’s no

download

url from NFS so to speak (NFS limited to controller), so controller provides an http url, and hence the payload has to pass through it.