Hi all, I just started working with a PoC for Flin...
# troubleshooting
g
Hi all, I just started working with a PoC for Flink here. I created a simple application that processes events from Kinesis. But I'd like to enrich those events with data from an external database by using a SQL Lookup Join. However, the application fails if I try to use the
FOR SYSTEM_TIME AS OF
statement on Flink SQL. Error:
"Temporal table join currently only supports 'FOR SYSTEM_TIME AS OF' left table's time attribute field"
.
Copy code
Table joinedTable = sedona.sqlQuery(
"SELECT i.proc_time, i.customer_id, ca.area_id " +
        "FROM Input AS i " +
        // FIXME the lookup should be made on Postgres but it's not working using the 'FOR SYSTEM_TIME AS OF' statement
        "INNER JOIN CustomerArea FOR SYSTEM_TIME AS OF i.proc_time AS ca ON ca.customer_id = i.customer_id " +
        "WHERE ST_Intersects(i.geometry, ST_GeomFromWKB(ca.area_geometry))"
);
Without using that statement, the application runs but the external database is queried once (the referenced table gets traversed only once) and after finishing the query gives no more results (in fact the operator from the JDBC source is shown as `FINISHED`on the Flink Web UI, while other operators are still
RUNNING
). I created a discussion on StackOverflow also if this could help others with the same issue. Have you faced that issue already? Can someone give me any advices on how to solve that? Flink's version is 1.18.1.
πŸ› 1
h
Hi @Glauber Dantas I am also facing the same issue. Do you have any solution for it yet?
g
Not yet, looking for some solution. What I'm trying now is to register the tables in the Catalog instead of using the temporary views.
It indeed seems to be an issue on Flink (either a bug or a lack of clarity on documentation maybe?) But it seems to be required that the tables are registered on the catalog for the statement to work... something that doesn't happen when you convert from DataStream to Table through
tableEnv.createTemporaryView(...)
. By registering the Kinesis and Postgis both as Table using
tableEnv.createTable("my_path", TableDescriptor.forConnector("connector")...)
, everything work as expected and the Postgis database is pooled respecting the TTL etc.
I found a bug opened for Flink 1.13 related to this issue: https://issues.apache.org/jira/browse/FLINK-23307
h
How did you define the proc_time column? My problem was using the wrong function for the processing time attribute.
g
You can use the
PROCTIME()
function. e.g.
Copy code
Schema.newBuilder().columnByExpression("proctime", "PROCTIME()")
when defining the Table.
Would you mind upvoting this (old) bug on their JIRA? https://issues.apache.org/jira/browse/FLINK-23307
And on this new bug I opened as well: https://issues.apache.org/jira/browse/FLINK-35392
h
Yesterday, I realized that I wasn't using PROCTIME(), which was causing the issue. It works fine now that I am using PROCTIME(). Could you show the entire code? Maybe your code isn't matching something.