Hi Everyone, We are working on reducing the EBS volumes attached to pinot servers in our pinot cluster (deployed in AWS EKS) since they are over-provisioned. Is there any industry wide followed practice for such use cases.

if you are using k8s and all segments are backed by s3 deep store, then it's simple. • Just reduce PVC size in the statefulset to your desire size. • Delete the PV/PVC for that server pods. • Restart that pinot server pods and k8s will recreate new PV/PVCs with new size

Minor change since we do this. First step, cannot edit the statefulset volumeClaimTemplates (rejected by the API server), so can do a delete (with

--cascade=orphan

to keep the current pods running without a StatefulSet definition and recreate the StatefulSet CRD with the latest state that will restart pods according to your policy and update things up.

We have replication factor=1, we don't want any data loss or unavailability of servers during this time. Deleting and restarting new pod with reduced volume would cause availability issues during migration time. For consuming segments, in order to avoid data loss, we would have to force commit the segments and then trigger the deletion, during this time, there will be no consumption of data. Is there any way to avoid all these?

pinot keeps kafka consumer offset internally so there won't be data loss

Thanks Xiang So the process would be 1. Make changes in the template to reduce PVC size in the statefulset to our desire size. 2. Stop consumption to force commit and trigger upload to deepstore. 3. Delete PVC for the server pods to trigger the recreation of the pods with new template (with reduced size).

for 3, you need to delete both PVC and PV