Hi, I am Implementing user defined aggregate func...
# troubleshooting
k
Hi, I am Implementing user defined aggregate function and using the bellow Accumulator class, but getting bellow exception. the exception says Set is abstract and it is not allowed but
List
works fine
Copy code
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public class Accumulator {
    private double sum = 0;
    private long count = 0;
    private double minValue = Double.MAX_VALUE;
    private double maxValue = Double.MIN_VALUE;
    private List<Double> values = new ArrayList<>();
    private Set<String> buffer = new HashSet<>();
}
Copy code
Caused by: org.apache.flink.table.api.ValidationException: Could not extract a data type from 'class org.abs.security.utils.Accumulator' in generic class 'org.apache.flink.table.functions.AggregateFunction' in class org.abs.security.udfs.DistinctCount. Please pass the required data type manually or allow RAW types.
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:362)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:240)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRaw(DataTypeExtractor.java:218)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeWithClassContext(DataTypeExtractor.java:194)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractFromGeneric(DataTypeExtractor.java:124)
	at org.apache.flink.table.types.extraction.FunctionMappingExtractor.lambda$createGenericResultExtraction$13(FunctionMappingExtractor.java:478)
	at org.apache.flink.table.types.extraction.FunctionMappingExtractor.putExtractedResultMappings(FunctionMappingExtractor.java:319)
	at org.apache.flink.table.types.extraction.FunctionMappingExtractor.collectMethodMappings(FunctionMappingExtractor.java:269)
	at org.apache.flink.table.types.extraction.FunctionMappingExtractor.extractResultMappings(FunctionMappingExtractor.java:169)
	... 23 more
Caused by: org.apache.flink.table.api.ValidationException: Could not extract a data type from 'class org.abs.security.utils.Accumulator'. Interpreting it as a structured type was also not successful.
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:362)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:290)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:232)
	... 30 more
Caused by: org.apache.flink.table.api.ValidationException: Error in field 'buffer' of class 'org.abs.security.utils.Accumulator'.
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:362)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractStructuredTypeFields(DataTypeExtractor.java:539)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractStructuredType(DataTypeExtractor.java:513)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:288)
	... 31 more
Caused by: org.apache.flink.table.api.ValidationException: Could not extract a data type from 'java.util.Set<java.lang.String>' in generic class 'org.apache.flink.table.functions.AggregateFunction' in class org.abs.security.udfs.DistinctCount. Please pass the required data type manually or allow RAW types.
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:362)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:240)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRaw(DataTypeExtractor.java:218)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractStructuredTypeFields(DataTypeExtractor.java:536)
	... 33 more
Caused by: org.apache.flink.table.api.ValidationException: Could not extract a data type from 'java.util.Set<java.lang.String>'. Interpreting it as a structured type was also not successful.
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:362)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:290)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrRawWithTemplate(DataTypeExtractor.java:232)
	... 35 more
Caused by: org.apache.flink.table.api.ValidationException: Class 'java.util.Set' must not be abstract.
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:362)
	at org.apache.flink.table.types.extraction.ExtractionUtils.extractionError(ExtractionUtils.java:357)
	at org.apache.flink.table.types.extraction.ExtractionUtils.validateStructuredClass(ExtractionUtils.java:165)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractStructuredType(DataTypeExtractor.java:478)
	at org.apache.flink.table.types.extraction.DataTypeExtractor.extractDataTypeOrError(DataTypeExtractor.java:288)
	... 36 more
j
Currently, Flink's type inference does not support auto mapping
Set
to MULTISET (See https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/types/#data-type-extraction). It's recommended to use
DataTypeHint
as a workaround. (https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/functions/udfs/#type-inference)