Hi All We are using a MSQ query with a LEFT JOIN to update e Apache Druid #general

Hi All, We are using a MSQ query (with a LEFT JOIN...

11/18/2024, 2:50 PM

Hi All, We are using a MSQ query (with a LEFT JOIN) to update existing data in a table using the

sortMerge

join. When the query is used to update the data in a daily granularity table for an interval of 3 days, then it fails with the below error. Error: RowTooLarge: Encountered row that cannot fit in a single frame (max frame size = 8,000,000) The same query completes successfully when executed for each day separately or for an interval of 2 days (within the same 3 days as above). So this doesn't seem to be an issue with the size of a single row. The MSQ documentation that the max limit for a row size is 1MB. I couldn't find any details on the

maxFrameSize

configuration. What is this config? What is the 8,000,000 that appears in the error message - is it the number of rows? Is there any way to increase this value? Druid Version: 27.0.0. Thanks, AR.

11/18/2024, 3:26 PM

We found a MV column which has around 14k values. We think this might be the cause of the error in the MSQ update query. We are investigating further into this. Thanks, AR.

11/19/2024, 1:40 PM

The issue seems to be that NVL doesn't work as expected with MV columns. Query:

SELECT NVL(a.MV_COL, b.MV_COL) FROM TABLE_1 a LEFT JOIN TABLE_2 b ON a.COL = b.COL

In this case, instead of checking for NULL, the resulting column has the product of a.MV_COL and b.MV_COL. Not sure why this is happening but we resolved it by converting the MV col into an ARRAY and using MV_LENGTH to check for null rather than NVL.

Open in Slack

Previous Next