This message was deleted.

can you run the query with

"debug": true

in the query context, and check the broker logs for relevant stack trace

Hi Laksh, Broker request logging is enabled. Setting

"debug":true

didn't change the output in the logs. Below is the exception that I see. I don't see any stack trace. Do I need enable any debug logging in the broker? Exception: org.apache.druid.msq.sql.SqlTaskResource - Failed to handle query: a033f9b8-d5b7-4ec6-a02d-b8a49bac7554 (java.lang.RuntimeException: Error while applying rule DruidQueryRule(AGGREGATE), args [rel#123180:LogicalAggregate.NONE.[](input=RelSubset#123173,group={0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56,57, 58, 59, 60, 61, 62, 63, 64, 65, 68, 69},groups=[{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 68, 69}, {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65}],…. Query Context: { "useApproximateTopN": false, "useApproximateCountDistinct": false, "finalizeAggregations": true, "maxNumTasks": 5, "rowsPerSegment": 300000, "useCache": false, "populateCache": false, "debug": true } Thanks, AR.

can you share the query plans of both the queries that you are sending?

Hi Laksh, This I tried the EXPLAIN PLAN FOR command on the MSQ query (with the ARRAY_AGG aggregation column) which reads from S3 but it failed with the same exception. I am not sure if EXPLAIN PLAN works with MSQ queries so I changed it to use another Druid table instead. This Druid table contains the raw records of the same dataset (no rollup). But the EXPLAIN PLAN failed on this table as well with the same exception. So it looks like the query is failing in the planning phase itself. I removed the ARRAY_AGG aggregation and tried EXPLAIN PLAN for the MSQ query from S3. It failed with the error "must have an id". I tried specifying the param "queryId" & "sqlQueryId" in the query context but it still failed with the same exception. Is this an issue with the Druid version that we are running? Regards, AR.

i forgot to mention to fetch the explain plan for the query which gets executed successfully - with distinct, but less groupBy columns. If you can, I’d suggest updating to 28.0.0 or 29.0.0 (this has a patch release coming out, so you’d might need to update to that as well). Calcite version in 24.0.2 was super old which was updated in later versions. From the problem description, there seems to be some restriction in the number of grouping columns (if I had to guess, 64 seems to be the limit). I haven’t checked if that’s still present in the later versions, so it might still be present. Upgrading shouldn’t hurt though. You can try it locally before a more widespread deployment.

Hi Laksh, Tried EXPLAIN PLAN FOR with lesser number of columns keeping the ARRAY_AGG(DISTINCT.. ). It gives the same error "must have an id". I have added "sqlQueryId" to the query context but it still gives the same error. Is there some other "id" that it is looking for? If there is a limit on the number of grouping columns, then it should be documented somewhere as this is a major limitation. Regards, AR.

idt that you need to supply the id. Regarding the number of grouping columns - I am not sure if that’s a limitation or not, however since I don’t have 24.0.2 setup (and its an old version), it’d be difficult for me to debug

Thanks Laksh. Will see if I can upgrade to a newer version and test. Regards, AR.