Hi, I've noticed that using regex in TEXT_MATCH ends up getting different results from using REGEXP_LIKE. It appears that TEXT_MATCH sometimes misses data. Is this behavior expected?

Can you provide both the queries? Also @Atri Sharma to comment.

The syntax is typically different, so it would be great to see the queries

I made a minimal example where there is a string col, and I ingested the string

a.1

: This returns the col correctly:

select distinct(col) from test_regex
where REGEXP_LIKE(col, 'a.1')

However this returns nothing:

select distinct(col) from test_regex
where TEXT_MATCH(col, '/a.1/')

Did you try escaping the period?

I tried escaping the period in the regex string and it does nothing

Can you share the exact query that did not work?

@Atri Sharma ^^

Let me try this locally and revert

based on my brief debugging, I think this issue is because Pinot uses the StandardAnalyzer for the lucene index: https://github.com/apache/pinot/blob/088da3f8c2f7077c51b7d8531b7b967ad1cf58c6/pino[…]/local/realtime/impl/invertedindex/RealtimeLuceneTextIndex.java From my understanding, the StandardAnalyzer doesn't support special characters

It is indeed an analysis problem - it would be solved if we used Whitespace Analyzer. I am wondering if its worthwhile to allow users to specify which type of analyzer they wish to use while creating a text field. Thoughts @Mayank?

Yeah. Let’s file an issue on GH and discuss there?

Thanks