https://pinot.apache.org/ logo
#getting-started
Title
# getting-started
a

Alice

04/11/2022, 11:47 AM
Hi team, if I customize a new plugin and want to deploy it in Kubernetes, what should I do if I don’t want to build a new image?
m

Mayank

04/12/2022, 4:49 PM
May I ask what plugin do you have in mind?
a

Alice

04/13/2022, 12:56 AM
As for as I know, Pinot can transform an array under a json key to multiply records. But I want to get multiply records from one json object with multiply nested keys under a specific key.
@User, I think I’ll follow the Flattening part of the doc to write a new Decoder to transform one recode to multiply records. And any suggestion on deploying customized plugins? 😃
m

Mayank

04/13/2022, 3:03 AM
If you implement it as a
@scalarFunction
and drop the jar in class path, I think it should work. @User to confirm
j

Jackie

04/13/2022, 3:23 AM
Transforming one json value to multiple records cannot be achieved with transform (scalar) function. It has to be implemented as a record transformer. I don't think it is pluggable right now, but we should be able to make it pluggable. cc @User for some guidance
m

Mayank

04/13/2022, 3:24 AM
Thanks @User. For other custom functions that can be implemented as scalar functions, do they just need to be in class path to be picked up?
a

Alice

04/13/2022, 3:32 AM
@User Thanks. If I understand it correctly, it can be achieved by writing a decoder similar to this existing one, and it’s pluggable. right?
m

Mayank

04/13/2022, 3:33 AM
I think Jackie mentioned above that Record transformer is not pluggable right now.
a

Alice

04/13/2022, 3:36 AM
@User yes, I tried this way and it’s not pluggable. So, if I want it’s pluggable, I can try writing a new decoder?
j

Jackie

04/13/2022, 3:38 AM
@User Yes. To plug in a scalar function, simply annotate the function and put it in the class path
👍 1
a

Alice

04/13/2022, 3:39 AM
@User Is the recordtransformer in the above image the one you mean?
j

Jackie

04/13/2022, 3:40 AM
Yes, I was referring to the ComplexTypeTransformer
If decoder is pluggable, we can give it a try. A better way IMO is to make the record transformer pluggable so that you can plug in your own ComplexTypeTransformer
a

Alice

04/13/2022, 3:53 AM
@User Great. I like your idea to make the record transformer pluggable. At the moment, could you help us confirm if decoder is pluggable?
m

Mayank

04/13/2022, 3:54 AM
I checked the code, seems like decoder should be pluggable. It will need to be in the plugin-dir, and you can specify the name in stream config.
j

Jackie

04/13/2022, 3:55 AM
It is under the plugins module, so I believe it is pluggable
a

Alice

04/13/2022, 4:08 AM
So back to the question, any way I can use to deploy a new plugin to an existing cluster in Kubernets without building a new Pinot image? 😂
m

Mayank

04/13/2022, 4:09 AM
Yes, drop the jar in plugin-dir that is configured. Then use streamconfig to specify the name
👍 1