Hi All,
Trying to understand this error from Flink...
# troubleshootingd
David Bryson
09/12/2023, 1:08 AMHi All,
Trying to understand this error from Flink. I'm going from a table with "ticker, price, event_time" to a tumbling window and something about my aggregation is giving flink a hard time:
Copy code
py4j.protocol.Py4JJavaError: An error occurred while calling o8.executeSql.
: java.lang.AssertionError: Conversion to relational algebra failed to preserve datatypes:
validated type:
RecordType(VARCHAR(6) CHARACTER SET "UTF-16LE" ticker, FLOAT EXPR$0, TIMESTAMP(3) EXPR$1) NOT NULL
converted type:
RecordType(VARCHAR(6) CHARACTER SET "UTF-16LE" ticker, FLOAT EXPR$0, TIMESTAMP(3) NOT NULL EXPR$1) NOT NULL
rel: Copy code
165 tumble_window = Tumble.over(lit(10).seconds).on(col("event_time")).alias("w")
166 mean_ticker = input_table.window(tumble_window) \
167 .group_by(col("ticker"), col('w')) \
168 .select(col("ticker"), vector_mean(col('price')), col('w').end)David Bryson
09/12/2023, 1:15 AMSeems like it has something to do with watermarks on the destination table
David Bryson
09/12/2023, 4:44 AMI'm not sure if this is a bug or something weird with the way I'm defining the window
David Bryson
09/12/2023, 4:56 AMI really don't know where these null constraints are coming from
David Bryson
09/12/2023, 4:12 PMIf i put in
col('w').rowtimeDavid Bryson
09/15/2023, 9:37 PMAlso when I add a NOT NULL constraint in my sink it doesn't seem to register and i get errors on this stream
David Bryson
09/21/2023, 4:53 PMFor others who are interested, I ended up casting the type to TIMESTAMP
David Bryson
09/21/2023, 4:53 PMseems like kind of a weird work around but it got me unblocked