Hi, I'm using a percentileUDAF which computes perc...
# troubleshooting
s
Hi, I'm using a percentileUDAF which computes percentile by storing all the data in memory for the window. I know this is not the best way but I see very poor performance for the exact same UDAF in flink 1.12 vs flink 1.4/1.9. Is there any major change between the versions that could cause this issue? I have checked my configurations multiple times and nothing stands out in particular.
example
the exact same job running on flink 1.12 has horrible performance compared to legacy job running on flink 1.4/1.9 versions. I can confirm the issue is in the UDAF as replacing the UDAF with some built-in function like MAX causes the job too run fine.
UDAF Implementation for ref -
Copy code
package com.uber.athena.udf.common.udaf;

import com.uber.athena.athenax.vm.api.AthenaXAggregateFunction;
import java.util.ArrayList;
import java.util.Iterator;
import org.apache.commons.math3.stat.descriptive.rank.Percentile;

public class PercentileUDAF extends AthenaXAggregateFunction<Double, PercentileAccumulator> {
  public PercentileAccumulator createAccumulator() {
    return new PercentileAccumulator();
  }

  public Double getValue(PercentileAccumulator accumulator) {
    return getValue(accumulator, accumulator.percentile);
  }

  public Double getValue(PercentileAccumulator accumulator, int percentile) {
    double[] valuesArray = new double[accumulator.values.size()];
    for (int i = 0; i < accumulator.values.size(); i++) {
      valuesArray[i] = accumulator.values.get(i);
    }

    Percentile p = new Percentile(percentile);
    return p.evaluate(valuesArray);
  }

  // Processes the input values and update the provided accumulator instance
  public void accumulate(PercentileAccumulator accumulator, Double value, int percentile) {
    if (value == null) {
      return;
    }
    accumulator.values.add(value);
    accumulator.percentile = percentile;
  }

  // Retracts the input values from the accumulator instance
  public void retract(PercentileAccumulator accumulator, Double value) {
    accumulator.values.remove(value);
  }

  // Merges a group of accumulator instances into one accumulator instance
  public void merge(
      PercentileAccumulator accumulator, java.lang.Iterable<PercentileAccumulator> its) {
    Iterator<PercentileAccumulator> iterator = its.iterator();
    while (iterator.hasNext()) {
      PercentileAccumulator nextAccumulator = iterator.next();
      accumulator.values.addAll(nextAccumulator.values);
    }
  }

  public void resetAccumulator(PercentileAccumulator accumulator) {
    accumulator.values = new ArrayList<Double>();
  }
}
The bottleneck is in accumulate method which simply dumps the data into an ArrayList.
m
It would be good to validate with a Flink version that’s still supported in the community, preferably 1.17. Version 1.12 is 2,5 years old and there have been many changes already. Flink 1.4 and 1.9 are even older
Without validating this in a recent version, I’m afraid there’s no way for us to help you given the amount of code changes is super high
s
we're currently migrating our legacy Flink 1.4/1.9 platform jobs to a new platform that supports Flink 1.12 version. Once we have all the jobs migrated to 1.12 we should be able to upgrade the Flink version quicker down the line but it is difficult to do so currently. In most cases we find the flink 1.12 performs better but this is a very strange scenario. I'll spend more time to see if I can find something
We were able to fix this by using the
ListView
instead of
ArrayList
in the UDAF as suggested in docs - https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#mandatory-and-optional-methods-1.
ArrayList
seems to trigger frequent GCs which maybe causing the issue