https://pinot.apache.org/ logo
s

Sadim Nadeem

05/28/2021, 1:08 PM
suppose I created a realtime table with some kafka with ssl enabled configs .. now my kafka(Strimzi kafka) crashed or its ssl configs changed along with broker IP .. so can I point the same pinot table to the new kafka broker Ip and updated ssl configs without losing the old data .. Is backup & restore necessary while doing this or can directly update the table creation connfigs @Xiang Fu @Mayank cc: @Mohamed Sultan @Shailesh Jha @Mohamed Kashifuddin @Mohamed Hussain @Pugal
x

Xiang Fu

05/28/2021, 4:41 PM
if Kafka topic partition offsets are persisted then you are fine
s

Sadim Nadeem

05/28/2021, 4:53 PM
u mean I dont need to delete the table . .just to update the table configs where we give broekr url and ssl configs
and it will automatically start fetching from new kafka with same topic name and append to the exisitng data loaded into segments from old kafka
m

Mayank

05/28/2021, 4:55 PM
As Xiang mentioned, if the offsets are consistent in old vs new Kafka then you are good. For example, let's say Pinot consumed until offset xx in old kafka. Does the new Kafka have the same offset for those messages? If so, yes. If not, then you need to redo the table
s

Sadim Nadeem

05/28/2021, 4:58 PM
ok but as Xiang meantioned earlier that pinot stores offsets in zookeeper which comes with pinot helm installation .. so even if the kafka gets replaced .. the pinot setup zookeeper will be still there with the offsets . .so then we dont need to delete the table since nothing changed in the zookeeper of pinot
for the same topic
m

Mayank

05/28/2021, 5:00 PM
Pinot does do that. What I am saying is if old and new Kafka are not consistent (eg message m1 and offset o1 in old kafka is not the same as message m2 in offset o1 in new Kafka), then Pinot won't know about it. It will consume from the offset it saved in ZK, but that offset may not be correct for new Kafka
s

Sadim Nadeem

05/28/2021, 5:06 PM
ok got it .. u mean the ordering of events in both kafka shuld exactly comply on a particular topic .. but since old kafka is lost.. and the new kafka will get only the latest data ..since old data is already consumed in pinot and the retention period of kafka was of only7 days.. so the same ordering of events cant be present on the new kafka .. it will have the latest data starting from the time it is created which is after old kafka crashed/its ssl config /broker url changed
m

Mayank

05/28/2021, 5:07 PM
I am also talking about offset. That number is not contiguous like 1, 2, 3, 4,... It is only monotonically increasing. So one can have 1, 3, 5... another one can have 2, 8, 9....
s

Sadim Nadeem

05/28/2021, 5:10 PM
these few highlighted configs in the table c reation script got changed
yes mayank .. offsets can be anything .. its not monotonically increasing
m

Mayank

05/28/2021, 5:15 PM
Ok, let me ask you this. Are you doing this in production, or just playing around?
s

Sadim Nadeem

05/28/2021, 5:15 PM
got ur point .. so in some way .. kafka is a single point of failure .. if kafka fails .. we have to take backup and restore only in new realtime table .. since i dont think its possible to create a new akfka instance having the same offsets of the same messages
m

Mayank

05/28/2021, 5:16 PM
If you have setup kafka with enough fault tolerance, then it should not fail
s

Sadim Nadeem

05/28/2021, 5:16 PM
we are about to get into prod .. thus need to ensure that whatever we are planning is rock solid ..
right mayank .. 100% agreed ..
m

Mayank

05/28/2021, 5:18 PM
👍