Hi, we noticed that for a segment on a table 2 out...
# troubleshootinge
Elon
03/13/2021, 3:30 AMHi, we noticed that for a segment on a table 2 out of 3 servers have the same data, but 1 of the servers has less data in the segment. External view == ideal state and in the cluster manager, when I click on the table it says it's in a "good" state. What would cause that? It's an offline table.
k
Kishore G
03/13/2021, 3:33 AM
Real-time segment or offline
e
Elon
03/13/2021, 3:34 AMoffline only
Elon
03/13/2021, 3:34 AMI looked in the data directory to confirm the segment directories and one of them differs from the other 2
k
Kishore G
03/13/2021, 3:34 AM
Why do say they have less data?
e
Elon
03/13/2021, 3:35 AMOnly one of them does (out of 3). But when I look at the cluster manager it says the table is in a good state
Elon
03/13/2021, 3:39 AMAnd ideal state == external view for all segments. All online, with the 3 servers for each segment (replication factor is 3).
Elon
03/13/2021, 3:39 AMUntitled
Elon
03/13/2021, 3:39 AMHere's the table def ^^
Elon
03/13/2021, 3:39 AMA lot of bloom filter columns, not sure if that affects anything? This is pinot 6
Elon
03/13/2021, 3:57 AMCould it be due to zookeeper being messed up? We had an issue with an istio deployment that killed zookeeper. It came back up but had to reread snapshots. I can check there
k
Kishore G
03/13/2021, 3:59 AM
I don’t think so.. how are saying that one of them has less data?
e
Elon
03/13/2021, 4:00 AMYes, when I do select count where $segmentName = ...
k
Kishore G
03/13/2021, 4:00 AM
Can you pate the results
e
Elon
03/13/2021, 4:01 AMHere's from the server with less data:
Copy code
-rw-r--r-- 1 root 1337 38270 Mar 12 00:26 columns.psf
-rw-r--r-- 1 root 1337 16 Mar 12 00:26 creation.meta
-rw-r--r-- 1 root 1337 25203 Mar 12 00:26 index_map
-rw-r--r-- 1 root 1337 104475 Mar 12 00:26 metadata.propertiesElon
03/13/2021, 4:01 AMAnd the other 2 servers have this:
Copy code
-rw-r--r-- 1 root 1337 829877 Mar 13 00:05 columns.psf
-rw-r--r-- 1 root 1337 16 Mar 13 00:05 creation.meta
-rw-r--r-- 1 root 1337 25694 Mar 13 00:05 index_map
-rw-r--r-- 1 root 1337 105129 Mar 13 00:05 metadata.propertiesk
Kishore G
03/13/2021, 4:03 AM
That’s bizarre
j
Jackie
03/13/2021, 6:05 AM@Elon Did you replaced this segment? I think what might happen is that one server somehow didn't receive the message of re-downloading the segment, thus still keeping the old segment. Restarting the server should be able to pick up the new segment