Apache Pinot

Has anyone had any issues scaling Pinot on Kubernetes with StatefulSets? Is this path generally suggested over VMs?

Yes, we use it extensively and is definitely suggested over VMs.

Thank you Mayank.
Pros:
• Cost savings
• Autohealing
• Easier to maintain 

Cons:
• Kubernetes learning curve
• Managing state on K8s not primary use case

Are there other items you would add to that?

Internally we have headwinds to getting a stateful service on K8s so want to understand full picture

Auto-scaling, operational ease (upgrades, rollbacks, restarts) are some of the others.

Agree that k8s has frictions for stateful systems (as compared to stateless)

<@U05E495JBQ8>
This <https://github.com/datainfrahq/pinot-control-plane-k8s|project> is built for running pinot on k8s. It manages pinot state on k8s.

<@UDQU92KBK> Just want to make sure my reading of this is correct <https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart>
&gt; *Note*: Specify *StorageClass* based on your cloud vendor. Don't mount a blob store (such as AzureFile, GoogleCloudStorage, or S3) as the data serving file system. Use only Amazon EBS/GCP Persistent Disk/Azure Disk-style disks.

So none of the data can be stored in S3 (schemas, table configs, segments, etc)? This seems odd...wonder why this limitation exists?

Above is for server storage. You can always save Zk snapshots on S3

What about controller and minion storage?