Hello everyone, I have a Pinot cluster with 3 controllers nodes and I've set a daily retention on my segments. However, I've noticed that when I check the size of the Deleted_Segments folder on each controller node, the size varies significantly. For example, one node has 33 GB while another has only 2 GB. I'm unsure why this occurred and would appreciate any insights or suggestions on how to address this issue. Thank you!

Are you not using deepstore for controller data dir? Controllers are supposed to share the same storage (something like deepstore or NFS).

I’m not using deepstore or NFS for controller data dir

That’s incorrect setup

There is no documentation in the official Pinot documentation regarding the usage of NFS or deep storage for the Pinot controller directory. As a user of Pinot, I found this to be an important missing piece of information that can cause confusion and errors when attempting to deploy a Pinot cluster using these storage options. I would like to request that the Pinot development team consider adding documentation on using NFS or deep storage for the Pinot controller directory to the official documentation. This would help users to understand the steps required to configure Pinot to use these storage options correctly and avoid any potential issues. Thank you for your consideration and dedication to improving the Pinot documentation.

Thanks for pointing this out. There’s some documentation here. Perhaps, as a new user it is not apparent enough, or not in the right place. Any recommendations on where we should put this? https://docs.pinot.apache.org/basics/components/table/segment/deep-store