https://pinot.apache.org/ logo
Channels
Powered by
e

Elon

03/13/2021, 3:30 AM
Hi, we noticed that for a segment on a table 2 out of 3 servers have the same data, but 1 of the servers has less data in the segment. External view == ideal state and in the cluster manager, when I click on the table it says it's in a "good" state. What would cause that? It's an offline table.
k

Kishore G

03/13/2021, 3:33 AM
Real-time segment or offline
e

Elon

03/13/2021, 3:34 AM
offline only
I looked in the data directory to confirm the segment directories and one of them differs from the other 2
k

Kishore G

03/13/2021, 3:34 AM
Why do say they have less data?
e

Elon

03/13/2021, 3:35 AM
Only one of them does (out of 3). But when I look at the cluster manager it says the table is in a good state
And ideal state == external view for all segments. All online, with the 3 servers for each segment (replication factor is 3).
Here's the table def ^^
A lot of bloom filter columns, not sure if that affects anything? This is pinot 6
Could it be due to zookeeper being messed up? We had an issue with an istio deployment that killed zookeeper. It came back up but had to reread snapshots. I can check there
k

Kishore G

03/13/2021, 3:59 AM
I don’t think so.. how are saying that one of them has less data?
e

Elon

03/13/2021, 4:00 AM
Yes, when I do select count where $segmentName = ...
k

Kishore G

03/13/2021, 4:00 AM
Can you pate the results
e

Elon

03/13/2021, 4:01 AM
Here's from the server with less data:
Copy code
-rw-r--r-- 1 root 1337  38270 Mar 12 00:26 columns.psf
-rw-r--r-- 1 root 1337     16 Mar 12 00:26 creation.meta
-rw-r--r-- 1 root 1337  25203 Mar 12 00:26 index_map
-rw-r--r-- 1 root 1337 104475 Mar 12 00:26 metadata.properties
And the other 2 servers have this:
Copy code
-rw-r--r-- 1 root 1337 829877 Mar 13 00:05 columns.psf
-rw-r--r-- 1 root 1337     16 Mar 13 00:05 creation.meta
-rw-r--r-- 1 root 1337  25694 Mar 13 00:05 index_map
-rw-r--r-- 1 root 1337 105129 Mar 13 00:05 metadata.properties
k

Kishore G

03/13/2021, 4:03 AM
That’s bizarre
j

Jackie

03/13/2021, 6:05 AM
@Elon Did you replaced this segment? I think what might happen is that one server somehow didn't receive the message of re-downloading the segment, thus still keeping the old segment. Restarting the server should be able to pick up the new segment
Powered by