<!here> how can we increase pinot ingestion throughput if we are consuming messages from kafka topic . .say the output topic have only 1 partition .. will increasing partition of the kafka topic from where pinot is consuming automatically increase pinot throughput ... means pinot is smart enough to increase the no of consumers consuming from that topic and storing into tables/segments ..

Hi @Mayank Glad to say, He is also my team mate who is working on data engineering stuffs.

pinot will handle Kafka topic expansion

@Mayank regarding the discussion u had with @Mohamed Sultan.. why we need to create a new pinot cluster and restore the backup there.. so the use case is like .. suppose my gcp service account is changed and i need to migrate to a new gcp vpc .. then how can we restore the pinot backup from one cluster to another pinot cluster @Mohamed Sultan @Pugal @Mohamed Kashifuddin @Shailesh Jha please post what is the blocker we faced in restore

@Sadim Nadeem if you can copy data from old to new vpc then you can simply point the new cluster to the new vpc.

sure mayank .. ofcourse pointing two clusters to same tables will cause issue .. but restoring should not be a blocker @Mohamed Sultan

Yes, restore works.

@Xiang Fu @Mayank where will pinot store the kafka topic offsets(Checkpointing) while consuming messages from kafka .. since if pinot restarts .. then it should start consuming from the last processed message .. to ensure atleast once processing of all the messages published on the kafka topic from where pinot table consumes

in segment metadata

Thanks @Xiang Fu .. actually the problem was before we were using only one partition on kafka topic and we were using low level kafka consumer .. but in few tables .. those inbuilt consumers of pinot seems to getting stuck occasionally and unable to consume msgs from topic even though new msgs/events are published on the topic .. thus thinking of increasing partitions on the kafka topic to ensure reliability , resiliency and higher throughput since as u said earlier that eventually pinot will also increase no of consumers as the no of partitions will increase for low level kafka consumer ..

this table conf is good

might be some garbage data may get published

ic, we recently fixed a bug in kafka consumer. null message could cause kafka consumer hanging there: https://github.com/apache/incubator-pinot/pull/6950

ok so we need to upgrade the pinot with helm version

You can try latest docker image

we are using Google Kubernetes Engine deployment of Pinot

Change pullPolicy to Always

Running Pinot in Kubernetes - Apache Pinot Docs https://docs.pinot.apache.org › kubernetes-quickstart 1. Start Pinot with Helm — Pinot repo has pre-packaged HelmCharts for Pinot and Presto. Helm Repo index file is here.

Then restart all Pinot pods

ok means even if zookeeper url is empty .. its not incorrect

seems like pinot is already comes with zookeper by default. @Xiang Fu can you guide us how the default zookeeper connects with pinot?

zk url will be auto filled when deploy the helm

ok so checkpoints/offsets are stored in this zookeeper only .. thanks

Yes

Thanks a lot @Xiang Fu... very grateful for that

CC: @Shailesh Jha