https://pinot.apache.org/ logo
#general
Title
# general
m

Monica

03/09/2022, 9:21 AM
Hey everyone, I found pinot text index only support standard analyzer, is there any plan to support custom analyzers, like elasticsearch ? Or could you give me some advice how to support it better if we do this feature?
m

Mayank

03/09/2022, 2:16 PM
Not at the moment. Could you describe the use case where you would need this in Pinot?
k

Ken Krugler

03/09/2022, 6:59 PM
@User we worked around this limitation by doing the analysis in a Flink workflow, and using the resulting terms in a multi-valued string field that we used for queries (filtering). It doesn’t do true phrases, but we generate both one and two-term strings, and we do the same analysis for the user query, so it (almost) eliminates any false positives.
m

Mayank

03/09/2022, 8:12 PM
Thanks @User, always appreciate you help.
k

Kishore G

03/10/2022, 1:07 AM
Thanks Ken. For my own understanding, whats the use case for custom analyzers.. It should be easy to make the analyzer pluggable.
m

Monica

03/10/2022, 3:03 AM
Thanks for your reply.We have some Chinese fields that standard analyzer isn't suitable.
k

Kishore G

03/10/2022, 3:16 AM
Got it.. feel free to contribute this feature.. we can help you
k

Ken Krugler

03/14/2022, 9:45 PM
Hi @User - we have a lot of language-specific analyzers (16 total).
k

Kishore G

03/14/2022, 10:12 PM
should be easy to make this configurable rt
m

Monica

03/17/2022, 9:43 AM
Thanks @User. I'd like to contribute this feature.My main idea is to configure each text index field with its own analyzer.The configuration of the analyzer is similar to elasticsearch's custom analyzer:
Copy code
"analyzer": {
        "my_custom_analyzer": {
          "type": "custom", 
          "tokenizer": "standard",
          "char_filter": [
            "html_strip"
          ],
          "filter": [
            "lowercase",
            "asciifolding"
          ]
        }
      }
I consider put this config in
fieldConfigList
.Then I will let this configuration to the reference classes, like
LuceneTextIndexCreator
,
RealtimeLuceneTextIndexReader
,etc.. I've support it in my own pinot feature.But there are some problems I don't know how to handle: 1.How to support custom analyzer which is written by users, like extending
org.apache.lucene.analysis.Analyzer
, where to put these classes in pinot? 2.Currently I'm breaking a lot of classes methods and constructors, like
IndexCreationContext,SegmentGeneratorConfig
in SPI,
LuceneTextIndexReader's constructor
etc..I guess I should check configurations as much as possible before instancing, and try not to pollute the code.For example, lucene's classes should only exist in
pinot-segment-local
module. Could you give me some advice on how to contribute this feature better? If you also think this feature makes sense,I will try to take it and open a new issue on pinot's repository of github. :)
k

Kishore G

03/17/2022, 12:53 PM
Looks good at high level.. let’s file an issue and continue the discussion there. @User would love to get your input here
k

Ken Krugler

03/17/2022, 4:44 PM
We created a similar JSON-based format for specifying the analysis chain in our Flink workflow. Format looks like:
Copy code
"text_en_similarity": {
                        "char_filters": [
                                "similarity_newline_remapper",
                                "similarity_punct_remapper"
                        ],
                        "tokenizer": "StandardTokenizerFactory",
                        "token_filters": [
                                "LowerCaseFilterFactory",
                                "EnglishPossessiveFilterFactory",
                                "currency_remapper",
                                "similarity_stemmer_en",
                                "similarity_word_delimiter",
                                "number_remapper",
                                "similarity_stemmer_en"
                        ]
                },
We used the
xxxFactory
names for things that are standard Lucene classes, and other names map to custom tokenizers or filters (either regular filters or char filters) which are then defined in subsequent sections of the JSON. Given what I understand of adding something like this to the configuration, I’d suggest having a new section that defines every required analysis chain, and then in text index field definitions you can optionally provide the name of the analyzer to use. Maybe that’s what @User was already proposing…
k

Kishore G

03/17/2022, 4:55 PM
my only suggestion would be to flatten everything if possible so that it get read as properties.. I would like Pinot to be able to pass these properties to the analyzer without trying to understand the semantics of each property
k

Ken Krugler

03/17/2022, 5:43 PM
Would it make sense to have a new
pinot-text-analyzer
plugin subdir? And then have a standard analyzer as the one plugin?
Also, the analyzers feel more like Pinot FS implementations. You want to be able to instantiate a named analyzer, which can often have a bunch of associated classes and data (stop words, stemming data, etc).
m

Monica

03/18/2022, 4:05 AM
Thanks for your suggestions. I guess firstly I could only support analyzer's configuration from standard Lucene classes(maybe next step we could add analyzer plugin). For configuration format, I guess it's seems easier that each text index field has its total own analyzer, so the configuration won't change current configuration structure and backward compatibility, though different fields may have duplicate definition of configuration.So the table config may seems like:
Copy code
{
	"tableName": "transcript_analyzer_1",
    ...

	"fieldConfigList": [{
			"name": "text1",
			"encodingType": "RAW",
			"indexType": "TEXT",
			"analyzer": {
				"type": "custom",
                 ...
			}
		},
		{
			"name": "text2",
			"encodingType": "RAW",
			"indexType": "TEXT",

			"analyzer": {
				"type": "standard",
                 ...

			}
		}
	]
}
k

Kishore G

03/18/2022, 4:13 AM
Lgtm
I don’t think this warrants a plugin because Pinot does not understand this
Do you plan to add a custom or use one already in lucene
m

Monica

03/18/2022, 4:20 AM
I plan to use custom analyzer already in lucene, because it could support rich function with different configuration.
k

Kishore G

03/18/2022, 4:57 AM
Ok.. then this should be simple
k

Ken Krugler

03/18/2022, 3:07 PM
You also want to think about where supporting files live, and how they get accessed. Most users of Lucene analyzers wind up customizing behavior (stop words, protected words, etc). It would be hard to do this inside the schema, as you’d have to support JSON equivalent fields for all of the files, and re-work the analyzers to load from that versus from files found on the classpath, etc.