https://datahubproject.io logo
Join Slack
Powered by
hey all, we're seeing a weird issue in the recent ...
# troubleshoot
w
hey all, we're seeing a weird issue in the recent update where given a column 
ip_address
in a table, searching 
address
shows results but searching 
ip
has an empty result set. we have a large number of columns (and tables) that should match, so unsure what's going on
searching 
ip_addres
yields no results either, while 
ip_address
does
b
hey Tim! do you know if the behavior is different than how it previously worked? or is this just a recent observation? cc: @brainy-tent-14503
w
i believe its different, 
ip
is one of our common test search terms and we wouldve noticed this before
b
just curious: if you search 
ip_
does that show a different result?
w
no, i went character by character and it only showed once i had the full field name. 
addr
has no results, but 
address
shows the fields.
so it does seem to be weird behavior around the 
_
potentially
if i search 
acco
trying to find 
account
(another super common one) i see results for tables, but not columns
when i search 
account
i get 153 results
b
okay gotcha. we appreciate the feedback here! as you can imagine search is a tough one and something we're constantly trying to improve, so this is going to help us figure out what's going on and continue to improve upon it
w
yep! please let me know if there's any other info i can provide, its def impacting searchability of the data
b
Will need to look at column matching, the 
_
is delimited to 
ip
and 
address
the latter would be stemmed most likely. The 
ip
token is less than 3 characters and normally that token gets dropped. The full token should be preserved as well 
ip_address
and would like an example urn here if possible. I am interested in the column urns for one example, if you have it?
w
sure, sec
Copy code
urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:glue,database_name.table_name_here,PROD),[version=2.0].[type=string].ip_address)
b
Interestingly the column urns are not involved with dataset search. It seems to rely on the 
fieldPaths
or the string 
[version=2.0].[type=string].ip_address
specifically. Let me make sure this is included and tokenized correctly.
w
if it helps, given a column name 
BlockDeviceMappings
a search for 
DeviceMappings
returns no results
so it seems like its only matching full tokens rather than allowing matching of any part of the token
b
Field path like this is tokenized into the following, not great but at least parts of it make sense. I would expect 
address
and 
ip_address
to return results on that first example, but not 
ip
by itself.
Copy code
"[version=2", "version", "[type=string]", "type", "string", "ip_address", "address"
Let me seen if the camel case it split
w
even if it's all lowercase, shouldnt a partial match in the string work?
b
the latter example is being stemmed to 
blockdevicemap
and also lowercased
i would also expect 
DeviceMappings
to be stemmed similarly to 
devicemap
, lets see
w
did the search change from 
match_phrase
to 
match_phrase_prefix
or 
match_bool_prefix
?
b
The change was query_string to simple_query_string and match_phrase_prefix
Let me see what it was previously, I am not sure we had match_phrase
w
okay, 
query_string
can have params like 
fuzzy_prefix_length
that may also allow prefixes, and it's possible y'all were wrapping in wildcards as well or something. 
match_phrase_prefix
seems to match the behavior we're seeing
b
Looks like previously it was just 
query_string
with defaults, no adding in wildcards here this query builder.
That said, lets take a look at what we can improve here
w
interesting, i think it was working because it was splitting using the 
word_delimited
analyzer and allowed any order instead of a prefix
b
The 
fuzzy_prefix_length
can be adjusted on the 
simple_query_string
, those are defaults
We still use the 
word_delimited
analyzer
Might be slightly different though, certainly we added synonyms and stemming. (and other adjustments)
w
and the prefix requirement as well is important IMO
b
Or a combination of the prefix and non-prefix phrase match
The prefix was specifically added more for boosting the results rather than general recall
w
makes sense, earlier = more important
b
How important do you think supporting < 3 character searches is? Specifically 
ip
? I might need to add something like a syn to get it over the limit, like 
ip
=> 
ip_
- kind of a unique 2 char term I think, but interested in your thoughts.
w
middling, for my specific use case 
ip
specifically is fairly important because it can appear as either 
ip_address
, 
ipaddress
, 
source_ip_cidr
, etc
b
ok, let me create a test case around that specifically.
w
so when we have someone asking "where do we have IPs in our data" they would search 
ip
expecting all of those types of combinations to appear if that makes sense
b
yeah, so maybe a couple syns here. 
ip_address
, 
ip
, 
ip_address
are essentially the same. The 
source_ip_cidr
variation is a bit less so and I imagine 
destination_ip
and variations on that are possible
w
yep exactly
and technically the underscores could or could not exist in all of those as well, which i know makes it more fun....
appreciate your help with this btw, it's one of the primary use cases we evaluated when deciding on catalog options, so its unfortunately something we noticed pretty quickly. if there's anything at all i can do to help, dont hesitate to ask
b
Thank you for calling this out. I definitely think this is important to get correct and also to add to our tests to ensure things like this are not broken in the futurre.
What do you think about wildcard support for this use case. Would a search query like 
*ip*
be a problem or does it have to work with 
ip
?
w
are wildcards a one-or-more or a zero-or-more match? i.e in regex terms a 
*
or 
+
?
b
zero or more I believe
It doesn’t work today, but just looking at possible solutions.
w
if zero or more i think that's fine, though id suggest an update to the zero results blank state to mention the usage of wildcards as it wouldnt be super intuitive to someone just trying search for the first time. that's my initial thoughts at least.
or maybe automatic addition of wildcards to queries under 3 characters in length? though that could be confusing. maybe a sitewide option to enable/disable automatic wildcard addition to tokens if thats possible?
b
The 
query_string
we used to have supported this by default. Definitely needs documentation
w
makes sense - i had a feeling it was something about match_phrase_prefix from a prior gig i had where we were trying to tune ES performance but still had to have mid-string matching. it's not a fun problem, especially if you cant use text fields or whatever that allow it for up to 4k characters but not after that. 😂
b
There will be a new PR which includes that. I am working out bugs internally at the moment, but the plan is to get that up and merged prior to a release next week. This would theoretically work to make the double wildcard query work 
/q fieldPaths:*ip*
however it will pickup like 
shipping
and other false positives.
w
amazing, thank you! i think if someone searched 
*ip*
then 
shipping
would be a positive result!
6 Views
Open in Slack
PreviousNext
Powered by