Question about tiered storage+deep storage. We want to use pinot to query mostly fresh data (last week or so) and then store data long term on s3. We have deployed pinot in our k8s cluster using the helmchart and configuring the controller to backup to s3 seems to work. If I understand it correctly this is only to backup data which is also stored on local disks however and we want to keep the local disk usage low! Can we use tiered storage and configure one server to use s3 and another server to use local disks?

Hi Erik, this feature is only available in StarTree version of Pinot. https://www.startree.ai/blogs/introducing-tiered-storage

Oh! So these pages in the docs https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot and https://docs.pinot.apache.org/operators/operating-pinot/tiered-storage are not the same as that startree feature?

The concept is the same.. open source supports hdfs ssd, ebs etc as tiers

@User What about using S3 for deep storage (not tiered storage), is it also only supported in StarTree version?

no, that is available in OSS version as well