https://pinot.apache.org/ logo
Join Slack
Powered by
Hi all, I am trying to understand the diff between...
# general
z
Hi all, I am trying to understand the diff between 
native text index
(link) and 
text index
(link)? Is it that 
text index
is mainly for BLOB datatype? For my use case, my col is not as large as a BLOB(e.g., url), but I would like to support fuzzy search, prefix/suffix or wildcard match. Should I go with 
native text index
or 
text index
?
m
These are just separate implementations, search functionality wise they should be equivalent. cc: @Atri Sharma
a
They are the same -- just that native text indices are designed to be more disk space optimized, is absolute real time searchable (at the cost of heap memory) and are targeted for prefix/suffix and regex matches (so are faster). If you need a larger set of search capability (fuzzy), you should use lucene text indices
z
it seems there is a regression with version 0.12.1. previously I am able to do 
TEXT_CONTAINS (indexedString0, '.*bar')
but now it throws error -
Copy code
[
  {
    "message": "QueryExecutionError:\njava.lang.RuntimeException: Caught exception while running query: .*bar\n\tat org.apache.pinot.segment.local.segment.index.readers.text.NativeTextIndexReader.getDocIds(NativeTextIndexReader.java:111)\n\tat org.apache.pinot.core.operator.filter.TextContainsFilterOperator.getNextBlock(TextContainsFilterOperator.java:52)\n\tat org.apache.pinot.core.operator.filter.TextContainsFilterOperator.getNextBlock(TextContainsFilterOperator.java:37)\n\tat org.apache.pinot.core.operator.BaseOperator.nextBlock(BaseOperator.java:43)\n...\nCaused by: java.lang.NullPointerException",
    "errorCode": 200
  }
]
Strangely, 
TEXT_CONTAINS (indexedString0, 'bar.*')
works fine.
Note that TEXT_CONTAINS supports standard regex patterns (as used by LIKE in SQL Standard).
also according to the doc, I have been trying to use regex for matching rows(for example, 
TEXT_CONTAINS ("indexedString0", "%inventory%")
) but it doesn’t returns anything. could you provide a few examples about how to regex please?
@Atri Sharma could you help take a look please? 🙏
a
Ack, looking.
z
Hi is there any luck?
a
Trying to replicate the issue
Just checking, does it work for 0.12?
z
Yeah it does
a
So something broke between the versions
z
curious if you can reproduce the issue?
Also AFAIK, this is for word matching( so “bar.*” only matches “bar-another-word” but not “ibar-another-word”). do you know how I can match the latter string?
a
Add a . * before v
z
For posterity, 
LIKE ("indexedString0", '%bar%')
works great to match “css-sidebar”, “customize-record-sidebars”
I wonder what is the diff between LIKE and TEXT_CONTAINS? Is one more performant than the other?
a
It is weird that something changed for you in 12.1
I will give an update by Saturday to understand what broke
z
more update: • this is not continuously failing though for example, it succeeds if i tried 
TEXT_CONTAINS (indexedString0, '.*bar.*')
or 
TEXT_CONTAINS (indexedString0, '.*foo.*')
but it fails when it is 
'.*man.*'
or 
'.*html.*'
it seems not because there is no matching because i am able to find matches using 
indexedString0 LIKE '%html%'
happy to jump on a call if it helps
a
That would help. Can you please find some time over the weekend? I am in IST
z
how about Jun 17 7pm IST?
also created an issue for tracking: https://github.com/apache/pinot/issues/10934
👋 any luck with the investigation?
a
I missed your message. I will need some more context for debugging so a call will help
z
I am available now. Does it work for you?
a
I have a call. How about same time tomorrow?
z
yes it works. thanks.
👋 I will be available for the next 2h. let me know when you are ready.
@Atri Sharma let me know if we are still going to meet today
@Atri Sharma @Mayank sorry for the nudge. but this has been critical to our roadmap this year. is there anything I can help with for debugging?
m
Perhaps you can try lucene based to unblock your self? @Atri Sharma ^^
z
is there any doc around this? just remove the 
native
part?
a
Yes. Or can you use 0.12? I checked and it works there
z
Are you able to reproduce with 0.12.1? I wonder if it is related to the new data we just ingested not because of the upgrade.
it is not a feasible option for us to downgrade since there is another fix coming with 0.12.1 that we want to include
a
I am not able to reproduce this even with 0.12.1
Hence the concern
@Zhuangda Z have you tried enabling this for your existing segments? I feel it might be due to your new data?
z
I will look into this config in a bit. can you help me understand how the data can cause this NPE?
a
I am debugging that, it’s definitely an issue, but can be a data issue which is not being handled properly
@Zhuangda Z Lets talk tomorrow. I have freed up my calendar for this. Do you have some data that we can use to repro in a debugger?
z
Sounds good. What time works for your the best? All our data resides in our QA cluster but we can try access the server logs together to see if that yields some fruit
a
9 pm IST works well
z
cool see you then
a
Joining now
z
Action Items from the meeting: • Atri to share the segment file prefix for native text index • Zhuangda to get the file for analysis • Atri to share a patch for adding more debugging logs for index writer • Zhuangda to do a fresh ingestion to ensure segment is built correctly
m
Thanks both on driving this 🙏
z
@Atri Sharma could you help share the two listed items please?
a
Ack
There is a potential bug that was found in the area -- I am fixing it so that you can try with the correct build. Give me a few hours
🙏 1
z
Just check in to see how does the fixing go
a
Testing it. Seems to be going fine till now
z
Is there an open PR I can track the status? It would be great if you can link to this issue? https://github.com/apache/pinot/issues/10934
a
Ack
Can we get onto a quick call? Does tomorrow work?
z
sorry can’t make it tomorrow. How about today?
a
I am actually out today evening. NVM, let’s talk on slack
z
sure
This has become concerning to our team. We are evaluating if this will block our release later this year. We’d like to get your sense of your fix and verify it with our fork asap. Appreciate your help and understanding.
a
Ack. This is my top priority and I will get this sorted before Monday for sure. I am eager to see text indices at such scale :)
👍 1
z
We are increasingly concerned by this issue as neither it has a clear diagnosis nor we clients have a way to provide more debugging information. We understand this is a open source project and it’s still in its early stage but it looks a mechanism to quickly iterate on critical features is lacking unfortunately. cc: @Mayank
a
Did you try with the patch?
z
What patch?
a
We figured the issue was with the wrong store .
z
We haven’t heard anything since our last conversation
a
Sorry, I think I messed up. So, summary: I was able to reproduce the issue and it was coming since we were using a wrong byte store in the FST data structure. I have the fix ready -- but it requires some API changes and that is what is causing the delay (since the same data structures are also used by another index, the FST index). I have a patch with some hacks to work for the text index while I fix the overall tests for the other index. You can deploy it today itself and see if it fixes the issue. Apologies for the delay, it has been a crazy week at office, and the FST issue had a bigger blast radius than expected
👍 1
z
Do I need to reingest all the data or a server update would suffice?
a
You will need to update the Pinot binaries with a custom build (with the patch applied)
Open in Slack
PreviousNext
Powered by