https://pinot.apache.org/ logo
Join Slack
Powered by
We run a realtime table `table1` with fields `X` i...
# troubleshooting
m
We run a realtime table 
table1
 with fields 
X
 in 
upsert
mode. When a new field 
Y
 is added to the schema, a simple query
Copy code
select * from table1 limit 10
  in the Pinot explorer will return the following error:
Copy code
[
  {
    "message": "MergeResponseError:\nData schema mismatch between merged block:  [X(DOUBLE)] and block to merge:  [X(DOUBLE),Y(DOUBLE)], drop block to merge",
    "errorCode": 500
  }
]
However, the following query would work as expected
Copy code
select * from table1 limit 10 option (skipUpsert=True)
Has anyone seen this before?
m
@Jackie @Yupeng Fu ^^
j
Have you reloaded the table after adding the column?
m
@Jackie Hi again! Here is my response to you on Github
Yes we did. We don’t see this error with a regular table. This only manifests when 
upsert
 is on.
That said, maybe we are doing the reload incorrectly? Can you let us know the right way to do it?
j
The current reload has limited support on consuming segment, which might not work properly with upsert enabled. Restarting the servers will recreate the consuming segment, and apply the new schema.
@Yupeng Fu Have you run into this issue before?
m
we are way past creating a consuming segment already
could you also advise on the right way to restart the server so that a consuming segment can be closed and persisted to disk?
j
It does not need to be persisted to disk. We want to destroy the in-memory one and re-create one when the server starts
Send a signal to kill the server should be okay
m
Okay so an offset is not committed to Kafka to a segment is built? Good to know
Back to the main question, there are already numerous new segments being created since the new filed was added. So the consuming segment should not be an issue
j
During consumption, there is nothing persisted. Pinot uses low-level Kafka consumer, and maintains the committed segment offset in ZK. It does not commit the offset to Kafka
Since there are already new consuming segments created, can you try reload the segments again and see if the problem is resolved?
m
Already did last night. It didn’t work. Unless we did it wrong using the reload API
j
Do you have a lot of segments? If not, you may try this query to figure out which segment does not have the new added column: 
SELECT MAX(Y) FROM table GROUP BY $segmentName LIMIT 10000
m
Although we have thousands of segments but only 13 segments were retruned by the above query in an upsert enabled table and all the 13 segments had this field
j
The segments without this column won't be returned. You can also try 
SELECT DISTINCT $segmentName FROM table LIMIT 10000
to get all the segments
m
I guess upsert enabled tables are unique. You will need to do 
SELECT MAX(Y) FROM table GROUP BY $segmentName LIMIT 100000 option (skipUpsert=true)
will return all the segments
alright, so there are 15k segments in total and about 3k don’t have this field. Guess we were doing reload incorrectly? Could you help advise on the right way to do it? Further, would you be able to explain why we don’t encounter this error if we do 
select * from table1 limit 10 option (skipUpsert=True)
?
y
That’s surprising because the upsert impl is decoupled from the schema evolution
It would be helpful to see if there’s err log in the server log
j
We actually want to find all segments with valid docs after upsert, and 
SELECT DISTINCT $segmentName FROM table LIMIT 10000
should give you that
select * from table1 limit 10
will read the first 10 valid docs, and with upsert enabled, it might need to read more than 1 segment, thus cause merge conflict
m
@Yupeng Fu there is actually no error in the logs. also FYI, if we query the issue via Trino, no such error is returned.
@Jackie i answered that. 15k in total
j
I mean with upsert on
m
yes
j
What? With upsert off you got 15k right?
With upsert on you should get less
m
I got the same count
j
How about 
SELECT COUNT(*) FROM table GROUP BY $segmentName LIMIT 10000
?
Anyway, I think reload is somehow not done correctly. Let's reload again and see if the issue is fixed
m
`SELECT COUNT(*) FROM table GROUP BY $segmentName LIMIT 10000 returns`15k rows
@Jackie again, mind telling us the correct way to reload?
j
You can use the cluster manager UI to do the reload
Or use the rest API: 
POST /tables/{tableName}/segments/reload
m
Isn’t this method deprecated already in Pinot 0.8.0? We have been using 
POST /segments/{tableName}/reload
. Also since we are at this, what is the impact of a 
reload
? Will a segment under 
reload
become unavailable for query?
the reload has finished, but still there are only 12k segments having this field…and this doesn’t explain why querying via trino works…
BTW, both 
SELECT MAX(Y) FROM table GROUP BY $segmentName LIMIT 100000
and 
SELECT MAX(X) FROM table GROUP BY $segmentName LIMIT 100000
return 13 segments…so I suppose when the table is queried, only these 13 segments are accessed and they all have both fields X and Y. Following the logic, such an error should not have been thrown out.
j
Can you please check if the 13 segments are all consuming segments? These 2 queries should scan all the existing segments with valid docs
m
only 1 is. the queries won’t scan all existing segments in upsert mode. At least that’s how upsert works as per the design doc
j
For upsert table, the segment won't be returned iff all the documents within the segment is overwritten (becomes invalid).
SELECT COUNT(*) FROM table GROUP BY $segmentName LIMIT 100000
and 
SELECT DISTINCT $segmentName FROM table LIMIT 10000
should return the same segments. Not sure how you got different results for these 2 queries
Open in Slack
PreviousNext
Powered by