Hi I have a question. If the controller is not able to flush data to s3, will the Pinot server keep consuming data from Kafka? In our case, it was not consuming. When we fixed the s3 access issue and pinot was able to flush data to s3, it started consuming data.

If I'm reading the code correctly, the consumer thread does block while it's trying to commit to the deep store (s3) https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pi[…]ot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java There is a way to stop the server from sending segments to the controller for it to push to the deep store, but I don't think it'd help in this case as then the server would have got blocked trying to push to s3 instead. https://docs.pinot.apache.org/operators/operating-pinot/decoupling-controller-from-the-data-path

hmm. i am not sure i quite understood your question (and probably Mark too). did you mentioned you have s3 access issue ? several clarifications (if I understood correctly) 1. the code Mark shared is when consuming pinot-server commits a completed segment metadata to controller; it doesn't trigger a deep store upload -- it seems like updates segment metadata and tells controller it has been committed. 2. [UPDATE] deep store occurs depending on whether you configured split commit of not --> with split commit the server upload directly; otherwise the controller will receive the segment tarball and upload to deep store 3. deep store upload will also be validated again in https://github.com/apache/pinot/blob/91d2a958945f0b5d862821a216b0cba065be8461/pino[…]not/controller/validation/RealtimeSegmentValidationManager.java. it is a controller periodic task.

@Rong R Yes,as we checked the controller, we saw that the segment push to s3 was getting 404 from s3. When we fixed that and segments were pushed to s3, we could see all the messages in the controller in the web UI

Got it. So your concern was that segment upload were timing out and during that time the servers are not actively consuming Kafka. And this goes a way once controller have s3 access. yes?

Yes, to make it more clear, the timelines 1. we had few tables and there were few segments 2. we set up deep store in s3 3. New segments get created 4. We start noticing, few records are missing by querying in controller web UI 5. We push more data to validate 6. 2-3 records out of 6-7 are not visible 7. we check controller logs and found segments are not pushed to s3 8. Fixed the s3 issue 9. segments get pushed to s3 10. We are now able to see all records

You can configure direct push to S3 (instead of via controller) as pointed out before. Your consumption will continue, and the push will happen automatically when S3 comes back online.