I have a table that have the exact same segments from the controller API on two different clusters but the row counts for the table are different.

In the response it says all the servers responsed

also, we set up the cluster in a way where the controllers don't share a single filesystem. Could that be causing this?

You can also check external view via zooInspector

Yes both clusters have 253 segments that are online

If there is no partial response, then the only plausible reason I can think of is somehow the two clusters are seeing different data.

To narrow down, you can start adding time predicates (assuming you have time column) and verify each day’s data.

The way I verify the segments are identical is by using the get segment API to download all the segments from both clusters and compare each segment file content.

The way I am running into this problem is that I start with a cluster and download its segments. Then I post its segments to another cluster. Then I confirm both clusters now have the same segments. But cluster 2 has more data than the original cluster from the table row count. So it would seem some segments in the original cluster are not being used somehow. But the external view says all the segments are online.

Also, you mentioned that controllers do not share storage. All controllers within a cluster must share storage. Not sure how you upload the segments to the cluster, but if you use a vip, for example, segments could go to any controller. What are the servers configured with as the controller address?

@Subbu Subramaniam I counted the segments and the names across the two views, they are the same and all online.

@Mayank No I haven't touched the original cluster from which I downloaded the segments because it is our prod cluster. I did however upload segments from the prod cluster to two separate clusters. The counts for the two other clusters from the uploaded segments have the same row counts, which is higher than the row counts for the prod cluster. I wonder if prod is seeing all the segments correctly. It seems like the higher row counts that appear in the other two clusters and match are the correct numbers.

The comparisons I was talking about were between prod cluster and staging cluster. Initially I downloaded some segments from prod and uploaded to staging. For the same segments, prod reported less counts than staging. Then, I also uploaded the same segments to another cluster called dev. Dev and staging clusters report the same row counts. All three clusters have the same segments and they all show up as online in external view.

This leads me to think that prod cluster is the one with incorrect row counts.

Yes, seems like prod cluster is the incorrect one. But from all the info you provided, I am unable to think what might be causing the issue

Are all three offline only?

I suspect restarting the prod cluster will fix the issue just not sure what the root cause is.

Is it possible that the prod cluster has a realtime table with the same name?

@Andre Hsu what is your controller config for vipProtocol, vipHost, and vipPort? To provide you context, when a segment is uploaded, the controllers use these values to set the segment access URL on the segment metadata. The servers use the segment metadata to download the segment when they need to do so. The server's request to download can go to any controller (via the vip) and if that controller does not have the segment, then the server will never get the segment. In this case,the EXTERNALVIEW will be in ERROR state, though, so I am confused how things are working at all.

how do I check for vipProtocol, vipHost, and vipPort?

And also that you do have multiple controller hosts in prod?

It is possible that you ended up in a state where due to missing shared storage, the data seen by prod cluster is not the same as the data you are downloading from the prod cluster.

For example, different controllers have potentially different segments (with same name). And the prod servers are loading one copy of these differing segments, however, you are download a different copy with the same name.

we have five controllers currently in the cluster.

That would be my guess.

Maybe logging into the servers and seeing what segments they're actually seeing?

and compare those to the segments I downloaded from prod?

If so, just compare those in all 5 controllers and see if their size matches.

Also, segments on controller are tar.gz files, you can unzip them and

vim

the metadata.properties to find the

total.docs

in that segment.

I sshed into each controller in prod cluster, and went into /pinot_data/controller_data/tablename

I see that for the year_week that has issue, there are 253 segments and they are spread across each controller.

meaning each controller has a portion of the 253 segments, is that an issue?

But somehow, the servers are still able to download the segments from the correct controller?

I checked the total segments file size by summing up the segment sizes from the 5 controllers. The number is the same as the total segments size from downloading the segments and summing up the file sizes.

basically each controller has a subset of the total segments for the table and adding those up result in the same 253 segments as calling the download segment API for all 253 segments.

Is that set up correct? I think I found the issue, I must have missed the external view output the first time. One of the segments outputs ERROR.

I logged into the server that's supposed to have the segment and the segment is missing from the server just like you predicted. Maybe the server is not asking for the segment from the right controller since the segment is on one of the controllers.

@Subbu Subramaniam as you said, one of the segments actually showed up as ERROR. How do I check if the vipProtocol, vipHost, and vipPort are correct? It seems that the servers know which controllers have the segments it need but one segment.

What does it return for the segment in ERROR state?

It will be super helpful for you

In the meanwhile, you can also use the endpoint

controllerHost:controllerPort/tables/<tableName>/segments/<segmentName>

to access the segmentZKMetadata that has the download url.

I got zooinspector to work

{ "id":"dma_behavior_2018_10_2018_10_130" ,"simpleFields":{ "segment.crc":"577498001" ,"segment.creation.time":"1556344508517" ,"segment.end.time":"-1" ,"segment.index.version":"v3" ,"segment.name":"dma_behavior_2018_10_2018_10_130" ,"segment.offline.download.url":"http://controller-02:9000/segments/dma_behavior/dma_behavior_2018_10_2018_10_130" ,"segment.offline.push.time":"1556344540556" ,"segment.offline.refresh.time":"-9223372036854775808" ,"segment.start.time":"-1" ,"segment.table.name":"dma_behavior" ,"segment.time.unit":"null" ,"segment.total.docs":"179477" ,"segment.type":"OFFLINE" } ,"listFields":{ } ,"mapFields":{ } }

i believe the offline download url is correct and point to the right controller url. So it seems like it's set up in a way that server will ask the right controller for the segment?

Also is http://controller-url/segments/table_name/segment_name supposed to return code 404?

I can get the segment names if I omit the segment_name in the path though

ah I see why, it's because asking for specific segment is controller specific the way we set this up

This issue has been resolved. The set up environment with separate controller volumes may cause race conditions. We saw that there was a problematic segment that was in ERROR state and missing from the server. I will reupload that segment as a temporary fix and get the controllers to share volume maybe leverage deep storage.

Thanks for your help!