Hey team, we notice that pinot official doc mentioned this

⚠️ Enabling Groovy

Allowing execuatable Groovy in ingestion transformation can be a security vulnerability. If you would like to enable Groovy for ingestion, you can set the following controller config.

controller.disable.ingestion.groovy=false

If not set, Groovy for ingestion transformation is disabled by default.

Do you know if there is a best practice to use such feature 🙇 cc: @Neeraja Sridharan @Sai Tarun Tadakamalla

@Mayank Do you know who might be able to help us with this ^^ ?

Groovy is for fast prototyping of transforms. Once you know it works,, and it is a generic enough one, you are welcome to add it in the Apache Pinot source code. Alternatively, if this is an air-gapped deployment and you think the security threat does not apply to your deployment, you can enable it.

@Mayank thank you so much for your input! Sounds like groovy is good for the fast PoC but might not fit for secure production environment. A safer way should be UDF or contribute the stable function back to Pinot source code?

Yes, as long as it is a generic and reusable by others UDF

you can follow this video to build your own udf once you finished the poc with groovy

Thank you! Would it be possible to create a custom function/scalar UDF to generate HLL sketch as mentioned here?

that's possible as long as it's 1:1 mapping

Thanks @Xiang Fu! > that's possible as long as it's 1:1 mapping Sorry, I couldn't get this ^^. Did you mean generate HLL for each element in the array/multi-value column? We're planning to use this to create the UDF, and apply it as a row-level transformFunction as part of the table's transform config. https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/stream/cardinality/HyperLogLog.java

what I'm saying is that 1 row will become one HLL object

Yes, and then we plan to use

DISTINCTCOUNTHLL

on the column generated using this UDF row-level transform.

ack