Hello - we are exploring storing metrics at higher...
# general
s
Hello - we are exploring storing metrics at higher granularities by rolling up the data at lower granularities. Ex: 1s metrics rolled up and stored at 1 min granularity. Does pinot support percentile aggregations ?
m
Yes, Pinot supports percentile, as well as approximate percentile (using TDigest). For your case, since you are rolling up you might explore percentileTDigest as it is an additive function.
s
thank you will explorer that !
@User is it possible to support percentiles as part of pre-aggregation i.e. during ingestion ? The docs say only SUM is supported
m
Currently only sum is supported. Do you have a lot or rows that have some dimension values that will get aggregated?
s
yes we basically want to store latencies over a longer term. Ex: hourly, daily, weekly percentiles (95, 99, 99.99 etc)
m
You can still do that at read time?
s
we were hoping to not have to store raw data (at the lowest granularity) and compute them from the aggregates at higher granularities - something similar to this https://druid.apache.org/docs/latest/development/extensions-core/datasketches-quantiles.html
m
Do you mind filing a GH issue for this? If there’s already one (I think folks have asked), could you +1? I think it would be a good feature to have.
s
will do thanks
👍 1