I assume you are referring to Apache Datasketches ...
# troubleshooting
m
I assume you are referring to Apache Datasketches (Druid just uses it)? If so, contributing to code is preferred.
p
Learnt it last week - Druid has its own legacy libraries and then they also support using Apache DataSketch one via an extension. We are using Druid's legacy one to compute sketch. I understand contributing to code as preferred method but just want to understand if it can be done via UDF or could there be any potential issues?
Makes it easier for me to sell contributing 🙂
m
Hmm, I'd recommend upgrading to the Apache Datasketches instead of relying on a legacy one. UDFs are typically for stateless transform functions, this would be an aggregation function.
p
Upgrading isn't straightforward because migration is difficult with existing data
We dont have the raw data to re-create dataksetch state for historical data
m
I don't think it makes sense to take a dependency on a legacy algorithm from another project.
p
I agree in principle. Unfortunately I don't really have an option since I need the historical data. This is also why I am considering UDF approach
I can't convert historical data, and can't merge it with data using new / other datasketch algorithm
m
Yea, unfortunately, I can't think of a solution in this case. I don't think you can do UDF for aggregations.