I was exploring the text search index (text_match) : https://docs.pinot.apache.org/basics/indexing/text-search-support#text-parsing-and-tokenization As part of this, Is there a way to use a custom tokenizer or a way to configure it? As an example, if I have a text

<http://abc.pqr.xyz|abc.pqr.xyz>

and would like to use

.

as also my split term token along with

space

,

tab

. So, in this example, I have three terms -

abc

,

pqr

and

xyz

For what it’s worth, Elasticsearch

standard

tokenizer (which I believe is effectively a passthrough to Lucene

standard

tokenizer) returns a singular token for `abc.pqr.xyz`:

POST /_analyze
{
  "tokenizer": "standard",
  "text": "<http://abc.pqr.xyz|abc.pqr.xyz>"
}

Response:

{
  "tokens": [
    {
      "token": "<http://abc.pqr.xyz|abc.pqr.xyz>",
      "start_offset": 0,
      "end_offset": 11,
      "type": "<ALPHANUM>",
      "position": 0
    }
  ]
}

Whereas

letter

tokenizer yields what you’re after:

POST /_analyze
{
  "tokenizer": "letter",
  "text": "<http://abc.pqr.xyz|abc.pqr.xyz>"
}

{
  "tokens": [
    {
      "token": "abc",
      "start_offset": 0,
      "end_offset": 3,
      "type": "word",
      "position": 0
    },
    {
      "token": "pqr",
      "start_offset": 4,
      "end_offset": 7,
      "type": "word",
      "position": 1
    },
    {
      "token": "xyz",
      "start_offset": 8,
      "end_offset": 11,
      "type": "word",
      "position": 2
    }
  ]
}

@Mayank Any plan for supporting this in upcoming release to make it configurable?

I’d recommend filing a GH issue so we can take that into account.

GH issue: https://github.com/apache/pinot/issues/10374