I am trying to setup pinot deep store, Controller config is this , controller.data.dir=s3://shipment-pinot-dev/pinot-data/pinot-s3-dev/controller-data pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.region=us-west-2 pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.storage.factory.s3.accessKey=************************ pinot.controller.storage.factory.s3.secretKey=*************************** , The server config is this pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=us-west-2 pinot.server.segment.fetcher.protocols=file,http,s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.server.storage.factory.s3.accessKey=***************************** pinot.server.storage.factory.s3.secretKey=2q**************************** i restarted my controller , i have some tables created and they have data. I am seeing no data being pushed to my s3 and no logs reletaed to this . Anything that i need to add ????

As far as I know, the Server instances should be configured for using the local filesystem, and the Controller ones S3. This is how we configured it in YouGov and it's been working fine, at least until now. But I'd wait for the Pinot devs' answer to be sure.

do you see any errors in the controller or server logs?

i dont see any error in either controller or server . Find the attached helm release file for my. Let me know if anything to be added

For an offline table, any existing segments that you have will not automatically move to s3 after you set this config. New segments will go

Thanks to all our issue is resolved. We have few questions. 1. We can see data is going in S3 but how do we verify if the deep store is working fine or not? Any suggestions. 2. one more observation: We had 18 segments visible in the pinot controller dashboard. The segment in s3 is 8. On checking both the servers, we can see total 8 segments in both the servers under path /var/pinot/server/data/index/table_REALTIME in server. When we deleted one segment from the server directly and also from the pinot controller dashboard. The number of segments was reduced. can you specify if the deleted segments will be pulled from s3 deep store. We waited for some time but the number of segments did not return to 18.