Hi team, if I customize a new plugin and want to deploy it in Kubernetes, what should I do if I don’t want to build a new image?

May I ask what plugin do you have in mind?

As for as I know, Pinot can transform an array under a json key to multiply records. But I want to get multiply records from one json object with multiply nested keys under a specific key.

If you implement it as a

@scalarFunction

and drop the jar in class path, I think it should work. @User to confirm

Transforming one json value to multiple records cannot be achieved with transform (scalar) function. It has to be implemented as a record transformer. I don't think it is pluggable right now, but we should be able to make it pluggable. cc @User for some guidance

Thanks @User. For other custom functions that can be implemented as scalar functions, do they just need to be in class path to be picked up?

@User Thanks. If I understand it correctly, it can be achieved by writing a decoder similar to this existing one, and it’s pluggable. right?

I think Jackie mentioned above that Record transformer is not pluggable right now.

@User yes, I tried this way and it’s not pluggable. So, if I want it’s pluggable, I can try writing a new decoder?

@User Yes. To plug in a scalar function, simply annotate the function and put it in the class path

@User Is the recordtransformer in the above image the one you mean?

is this documented? If not, lets document: https://apache-pinot.slack.com/archives/C01H1S9J5BJ/p1649821139330959?thread_ts=1649677655.491089&cid=C01H1S9J5BJ

Yes, I was referring to the ComplexTypeTransformer

@User Great. I like your idea to make the record transformer pluggable. At the moment, could you help us confirm if decoder is pluggable?

I checked the code, seems like decoder should be pluggable. It will need to be in the plugin-dir, and you can specify the name in stream config.

It is under the plugins module, so I believe it is pluggable

So back to the question, any way I can use to deploy a new plugin to an existing cluster in Kubernets without building a new Pinot image? 😂

Yes, drop the jar in plugin-dir that is configured. Then use streamconfig to specify the name