Flink SQL + UDF vs DataStream API Hey Flinkers. W...
# random
j
Flink SQL + UDF vs DataStream API Hey Flinkers. While Flink SQL combined with custom UDFs provides a powerful and flexible environment for stream processing, I wonder if there are certain scenarios and types of logic that may be more challenging or impossible to implement solely with SQL and UDFs. From my experience, more than 90% of the use cases using Flink can be expressed with UDF and used in Flink SQL. What do you think?
j
To my understanding, Flink SQL doesn't support to use custom states and timers, which would be necessary for building complex stream processing apps.
g
you are both correct.. most use cases (unless something more sophisticated and low level is required in which case you might need to fall back to the datastream) can be implemented via sql and udfs
d
@Giannis Polyzos are there tradeoffs in performance?
g
For the most part i don't think there is any.. Especially for users new to flink, because the sql api is easy to use and tuning some jobs (for example mini-batch optimization etc.) is easy, it might be easier to start faster, as you don't have the human/developer factor i.e code quality, optimizing serde etc.
d
What about in the case of temporal joins? It’s just as fast to do this using Flink SQL? Like if you want to maintain an updated window of latest elements from the join or some type of table or view? Can it be done and is it just as performant as temporal join using Table API?
I did see there are some newer constructs described here: https://www.ververica.com/blog/flink-sql-recipe-window-top-n-and-continuous-top-n
How does this compare to a Table API approach?
As an example I can do this:
Copy code
WITH 
    JoinedData AS (
        SELECT 
            Orders.orderId, 
            Orders.productId, 
            Orders.orderTimestamp, 
            Products.productName, 
            Products.productTimestamp
        FROM 
            Orders
        LEFT JOIN 
            Products FOR SYSTEM_TIME AS OF Orders.orderTimestamp
        ON 
            Orders.productId = Products.id
    )
SELECT *
FROM JoinedData
WHERE ABS(TIMESTAMPDIFF(SECOND, orderTimestamp, productTimestamp)) <= 3600;
But that’s not continuously updating.
It’s a temporal join effectively but what about streaming updates for this?
I think the question is when you need to do a temporal join and want streaming updates within a window do you need to drop down into Table API or DataStream API to achieve this or are there equal or better methods in Flink SQL API?