Hi Pinot team! I'm currently encountering a weird behavior in text index where regex is not matching rows as expected. For example, my query contains

WHERE TEXT_MATCH(indexedString3, '/.*nav#primary-menu.*/')

and its not matching a row with

nav#primary-menu

. My theory this is due to that special character

#

and that standard analyzer breaks them up into tokens like

nav

and

primary-menu

and never keep them as a whole. Would be greatly helpful if someone could confirm my theory! Thanks in advance!

@Raghvendra Yadav ^^

@Soon Your theory is correct. Pinot's default text index uses Lucene's StandardAnalyzer (specifically CaseAwareStandardAnalyzer), which tokenizes using Unicode word boundaries. The # and - characters are treated as word boundaries, so nav#primary-menu is stored in the index as three separate tokens: nav, primary, menu . here is the quick short test to showcase the behavior. If you want to split on # you can use WhiteSpaceAnalyzer.

// StandardAnalyzer (Pinot's default) splits on '#' and '-'
  // "nav#primary-menu" → ["nav", "primary", "menu"]
  // Regex /.*nav#primary-menu.*/ has nothing to match → 0 results

  // WhitespaceAnalyzer keeps the full value as one token
  // "nav#primary-menu" → ["nav#primary-menu"]
  // Regex /.*nav#primary-menu.*/ matches → results returned
  @Test
  public void testSpecialCharTokenizationExplainsRegexBug() throws
  Exception {
      String text = "nav#primary-menu";

      List<String> expectedStandardTokens = new ArrayList<>();
      expectedStandardTokens.add("nav");
      expectedStandardTokens.add("primary");
      expectedStandardTokens.add("menu");

      List<String> expectedWhitespaceTokens = new ArrayList<>();
      expectedWhitespaceTokens.add("nav#primary-menu");

      try (CaseAwareStandardAnalyzer standardAnalyzer = new
  CaseAwareStandardAnalyzer()) {
          Assert.assertEquals(tokenize(standardAnalyzer, text),
  expectedStandardTokens);
      }

      try (WhitespaceAnalyzer whitespaceAnalyzer = new
  WhitespaceAnalyzer()) {
          Assert.assertEquals(tokenize(whitespaceAnalyzer, text),
  expectedWhitespaceTokens);
      }
  }

TY for the example as well! Could we have both analyzers so that both versions stay in pinot?

@Soon Analyzer is tied to Index and Index is per column. If you want two different text index with different analyzer one option is to create virtual derived column and enable text index with different analyzer.

to create virtual derived column and enable text index with different analyzer.

Gotcha, thank you. Will explore this path!