Hi. Is this an optimizer bug in Pinot where optimizer cannot recognize parentheses to group boolean clauses thus could not differentiate between:

text_match(col, 'abc') AND ( text_match(col, 'def') OR text_match(col, 'ghi') )

vs

( text_match(col, 'abc') AND text_match(col, 'def') ) OR text_match(col, 'ghi')

Both gets optimized into the same query

text_match(__mergedTextIndex,'abc AND def OR ghi'))

which seems to be the wrong behavior for search.

I believe the required syntax for composite text match is different according to the docs[1]. So your query might instead need to be:

text_match(col, '"abc" AND ("ghi" OR "def")')

[1] https://docs.pinot.apache.org/basics/indexing/text-search-support#composite-query-using-boolean-operators

So if we have multiple text_match clauses searching the same column, we expect the optimizer to combine them and have the potentially wrong but crash-proof default behavior rather than leaving them as multiple independent separate text_match queries?

Hmm, I agree that does look like a potential bug given that there’s no preservation of brackets. However it would be important to verify that the query does not preserve grouping, and not strictly the

explain

codepaths responsible for generating a string representation of the query

In lucene,

AND

operator takes precedence over

OR

operator. So

text_match(col, 'abc') AND ( text_match(col, 'def') OR text_match(col, 'ghi') )

when translated to

text_match(__mergedTextIndex,'abc AND def OR ghi'))

would definitely be wrong since the generated query would be

(abc AND def) OR ghi

rather than the intended

abc AND (def OR ghi)

What I mean is that there are distinct codepaths responsible for rendering

EXPLAIN

queries. There could be a bug in what’s shown in the EXPLAIN query, while the codepath for executing the query could still be correct. I don’t know if that’s the case, it could be a bug in query optimization like you said.

Okay, just Chatted with the person who wrote this optimizer logic. Didn't know my teammate wrote it. We have a fix available and testing will test it out soon.