I'm trying to do a workaround in pyflink UDAF to r...
# troubleshootingd
David Bryson
10/11/2023, 2:41 AMI'm trying to do a workaround in pyflink UDAF to return multiple rows of data. I'd like to return an Array of Rows which contain my data. When doing this I get an error that seems to indicate it's not possible:
ValueError: ROW is not supported to be used as the element type of ArrayType.
David Bryson
10/11/2023, 3:34 AMAdditionally I found this code:
https://github.com/apache/flink/blob/master/flink-python/pyflink/table/types.py#L2314-L2323
David Bryson
10/11/2023, 3:34 AMis there a work around here?
David Bryson
10/11/2023, 4:01 AMWhat I'm trying to accomplish is this:
• My data science team would like to process a time based window, and produce interpolated points as a result.
• There may be more than one point during the window that needs to be interpolated
• I'm ok with producing the data into a separate stream and joining it later
• I guess they are trying to treat the UDF as a black box and do all the data processing inside the box, then return all the results they want. This is making my life extremely painful in the flink world.
j
Jaehyeon Kim
10/11/2023, 4:36 AMNote sure why it's required to modify udaf. To my understanding, interpolated points may be returned after being concatenated by comma. Then we'd be able to apply udtf to unnest those?
d
David Bryson
10/11/2023, 4:37 AMWell my interpolated points would need two fields: time, value
David Bryson
10/11/2023, 4:39 AMSo wouldn’t I need to return an array of tuples?
j
Jaehyeon Kim
10/11/2023, 4:58 AMBy creating a UDF, we can have control over the way how to split it. For example, the example UDF splits string into a word and length of the word - https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/python/table/udfs/python_udfs/#table-functions
d
David Bryson
10/11/2023, 5:04 AMI don’t understand
j
Jaehyeon Kim
10/11/2023, 10:20 AMI may not fully understand your problem so that my suggestion may not make sense. The bottom line is, as long as records are properly concatenated, we can parse it with a udf.
d
David Bryson
10/11/2023, 2:06 PMAh, are you saying I could return:
Row(Array(timestamp), Array(signal))
David Bryson
10/11/2023, 5:40 PMok, I think this is what you were getting at and this does solve my issue
2 Views