Hi Pinot team. I want to know. What happens if onl...
# general
c
Hi Pinot team. I want to know. What happens if only one of the Pinot servers loses the Kafka offset? More specifically, I am curious about the case of using an Upsert table without a Retention setting. For example, let's say the Retention of Kafka Topic is 6 hours. Pinot's Consuming segment is flushed every 2 hours. One Pinot Server has been stopped for 7 hours. Pinot's segments are probably as follows. RunningServer #1 : realtimetable__0__0, realtime__0__1, realtime__0__2, realtime__0__3 (Consuming) RunningServer #2 : realtimetable__0__0, realtime__0__1, realtime__0__2, realtime__0__3 (Consuming) BadServer #3 : ??? What happens to the segments in this case? Will BadServer copy the segments? Or will there be data loss?
k
As long as you have another replica that was consuming and generating segments, server 3 will download the segments., have you tried restating server 3
c
@Kishore G Yes, I ran the test. The test results show that the data is lost. When server#3 is run, it regenerates the "realtime__0__3(Consuming)" segment, "resets" the "offset", and then starts consuming. I triggered the flush of the "realtime__0__3(Consuming)" segment with the "forceCommit" command. The rouCount of the realtime__0__3(Consuming) segment consumed by server#3 is smaller than server#2 and server#3. {I was able to confirm this by looking at the logs of each pino server communicating with the controller API. } (Because there was data deletion in Kafka due to the Kafka retention setting) Looking at the pino server log, server#3 becomes the winner segment and performs the build. (The table is an upsert table without retention and the completionMode is download.) The server#2 and server#3 download the segment. When I run a query with the condition $segmentName="realtime__0__3" as a debugging query, I can see that there is data loss. (There is no data that has disappeared from Kafka.)
@Kishore G The test is wrong. The "segment.flush.threshold.time" value of the table used for the test was smaller than Kafka's "retention" setting. If Server#3 was stopped for a long time, but Server#1 and Server#2 flushed the segment at a time interval smaller than Kafka's "retention", Server#3 would have downloaded the segment from deep-store. The latest data consumption would not have caused any loss even if the offset was lost.
It is peculiar that the replica that lost data (Server#3) becomes the winner even though there are two consuming segment replicas (Server#1, Server#2) that did not lose data.
Is there any algorithm to select a winner during the consumer segment flush process?
k
It compares the offset and in general they should all end in the same offset unless your flush threshold is time based