This message was deleted.
# generals
Slackbot
05/10/2023, 2:42 PMThis message was deleted.
a
Abhishek Agarwal
05/10/2023, 2:47 PMIt's addressed by https://github.com/apache/druid/pull/12357
Abhishek Agarwal
05/10/2023, 2:48 PMtry setting
inSubQueryThresholdg
Guang
05/10/2023, 2:51 PMActually, I saw that fix. But I also need to parse the query before sending it to Druid using the same calcite parser Druid uses.
a
Abhishek Agarwal
05/10/2023, 3:57 PMso its your parser taking time? may I ask why do you need to parse the query? once parsed, does druid get the sql query or the native json query?
g
Guang
05/10/2023, 4:06 PMYes, the parsing is taking quite some time ~40+ seconds. I need to parse the query because I need to introduce extra where clauses based on permissions. So, I parse the query to create SqlNodes then convert it back to sql and send it to druid.
b
Ben Krug
05/10/2023, 4:12 PMA workaround is to use *MV_OVERLAP*(myCol, ARRAY['myVal1', 'myVal2',...]). Not sure this will work forever, although it should imo. Planner doesn't try a million variations, just sends it on.
g
Guang
05/10/2023, 4:14 PMI tried
MV_CONTAINSMV_OVERLAPGuang
05/10/2023, 4:15 PMAlso
Copy code
select l from mytable
INNER JOIN (values ('a'), ('b') ... <thousands_of_values>) t(v) on mytable.l = t.vb
Ben Krug
05/10/2023, 4:25 PMNice, that's more intuitive, gtk, thank you!
g
Gian Merlino
05/10/2023, 7:22 PMyou could also use native queries (JSON) which is what SQL plans into, if you end up needing to bypass the SQL parser and planner
g
Guang
05/10/2023, 8:35 PMI was thinking about sending it as json queries but I think I read somewhere that using SQL is the preferred way, as well some optimizations are only available via sql. Plus, the redundant parsing by druid didn’t seem to matter much, at least for small queries.
Guang
05/10/2023, 8:50 PMFor some reason when I parse the query it’s a lot slower (40s) than when druid does it (<1s). My code
Copy code
SqlNode parseSql(final String query) throws SqlParseException {
final SqlParser parser = SqlParser.create(query, SqlParser
.configBuilder()
.setCaseSensitive(true)
.setUnquotedCasing(Casing.UNCHANGED)
.setQuotedCasing(Casing.UNCHANGED)
.setQuoting(Quoting.DOUBLE_QUOTE)
.setConformance(DruidConformance.instance())
.setParserFactory(new DruidSqlParserImplFactory()) // Custom sql parser factory
.build(););
return parser.parseStmt();
}g
Gian Merlino
05/10/2023, 10:55 PMyeah, it's true in general SQL is the preferred way
Gian Merlino
05/10/2023, 10:55 PMhmm you're saying Druid parses it a lot faster than Calcite does when you embed it in your own app?
Gian Merlino
05/10/2023, 10:55 PMSeems odd… do you have the same version of Calcite?
Gian Merlino
05/10/2023, 10:56 PMProfiling might be useful to hint at what's going on (yourkit or sjk flame graph)
g
Guang
05/17/2023, 6:28 PMI was missing a RelOptPlanner to optimize queries before doing the parsing. Adding that solved that issue. Thanks!
g
Gian Merlino
05/17/2023, 7:18 PMah ok! neat!