https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
a
I recall that planning can sometimes take more time when there is a large IN filter involved. and @Adarsh Sanjeev made some changes to optimize the same. there are a few context parameters that could be set to improve performance
v
Are the ids string or number?
a
Is the query timing out during planning? This was a known issue for IN clauses that contained ~10k value and there was a PR to help with this. (https://github.com/apache/druid/pull/12357) Could you try to set the 
inQueryThreshold
query to 0 to check if the query is faster? You can increase it to a higher appropriate value afterwards
d
hi @Adarsh Sanjeev sure i will set this parameter in query context and try again.
Hi @Vijay Narayanan product_id is string but i am passing the product_ids without single quote. Example: select LOOKUP(fe.product_id,'dim_desc') as manf_desc ,fe.sales_month,fe.sales_year,fe.sales_quarter,sum(fe.dist_total_revenue) from fact_extrapolated_revenue fe where __time >= TIME_PARSE ('2020-12-31T000000.000Z') AND __time < TIME_PARSE ('2022-11-30T000000.000Z') and fe.product_id IN ( 1000015,1000112,1000140) GROUP BY LOOKUP(fe.product_id,'dim_desc') ,fe.sales_month,fe.sales_year,fe.sales_quarter
v
the filter without quotes would be the bound filter (even for string fields). Putting the quotes changes it to a selector filter which will be faster. Please put the quotes and check
d
thank you @Vijay Narayanan. Putting the product_id in quotes really improved performance. Now the query is executing in 1.75 sec.
Should i also use 
inQueryThreshold ?
a
Yup. go ahead 🙂
👍 1
r
and maybe also https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1674326908487929 I think this can be fixed automatically by the planner, can't it?
b
Did that help/work? A hacky workaround is to try replacing 
myField in ("item1",..."item10001")
with 
*MV_OVERLAP*(myField, ARRAY["item1",..."item 10001"])
. Then calcite doesn't try to plan it, and sends it straight to druid, so you avoid the longer planning times. Hopefully soon planning will be improved.
r
I don't think I was affect by long planning, because I had a special case for list of ints where my query interpolation just rendered then as a sequence of ints (without using the ? ), then when I cast to string (both sides, just to be sure), I got this insane improvment in the query time and huge decrease in cpu usage
b
interesting, glad it improved!
Open in Slack
PreviousNext
Powered by