General question: Why is pinot broker/controller a statefulset in the helm chart? Those can be deployments isnt it? They dont have any persistence afaik.

Pinot is a distributed system. Data is one component, but there is also metadata (cluster state), and these components need persistent metadata from ZK. Also cc: @User in case of more comments.

Pinot controller can have local disk to serve as deep store for segment download, in that case, the registered download uri has to have fixed control pod name, so controller should be statefulset

I see. I thought metadata is only in zookeeper not persisted by the controller/broker

For Pinot broker, the default setup is all on default tenant, however, users have the ability to assign tenant for each broker and Pinot internal will maintain the mapping for which broker has which tables to serve.

Ya ofc..that makes sense