hey folks, another flink question. for simpler data pipelines in an organisation has anyone found a preference over Flink SQL vs using something like PyFlink? we're tempted around FlinkSQL due to everyone being familiar with it + having less 'power' in using Python and also the fact that there are no performance overheads as opposed to PyFlink. sadly, flink-java is not really an option here beyond maybe abstracting away some common functionality in a configuration driven approach.

My personal suggestion is always use Flink SQL, unless there is something quite sophisticated that requires lower level primitives

interesting 😄 can i ask why?

It’s simple, easy to use and understood by many different teams. Unless there is a use case that requires more fine grained control over time or state management, sql should be the way to go

that's a fair point! i'll do some research into how feasible this is.

Cdc and lakehouse use cases also become extremely simple.

can i ask why is this the case again sorry?

Flink cdc 2.0 + has gone lots of work to provide the best cdc support out there. Now combine it with lakehouse technologies like Apache Paimon (prev. Flink table store) you can easily implement easily data synchronisation and other use cases. Assume you have a database with 100, 1000 or more tables and you want to synchronise across different databases and cheap storage. You just write a single CREATE DATABASE AS statement and the framework handles everything automatically for u. For data engineers, analysts, scientists etc. that are familiar with sql, this is powerful.

thank you Giannis this is really insightful. i'll do some more reading/research!

It’s on my backlog, will be sharing more ~end September / early October around these. 😄

i'll keep my eyes peeled 👀

Tbh personally I have only used json and avro, but overall I haven’t seen something that indicates it’s hard. But a quick PoC should verify this

👍 i'll give this a look! thanks again

Pyflink also supports SQL and I have also sourced/sink data to protobuf using it.

i've been looking into it briefly and it seems a format of

protobuf

for a table connector is only available in 1.17 😕

I'm not following. Protobuf has been supported with table connectors for a while now.

'connector' = 'kafka',\

'property-version' = 'universal',\

'topic' = 'mytopic',\

'properties.bootstrap.servers' = 'kafka:9092',\

'scan.startup.mode' = 'earliest-offset',\

'format' = 'protobuf',\

'protobuf.message-class-name' = 'com.foo.Message',\

'protobuf.ignore-parse-errors' = 'false'\

huh ok, i found out from this jira ticket to add pb support it says 1.16

Yes..1.16.0, but the current release is 1.17.1. I am not sure why people would not choose pyflink/SQL for most basic usecases nowadays. Everyone knows python and SQL.

ah right sorry i forgot to clarify. we're using AWS managed flink which is pinned to version 1.15.x

AWS definitely doesn't support pyflink very well/at all. I was never able to get that to work using their managed services.

oh believe me i've been through that one 😆 our biggest flink pipeline in the company runs on PyFlink and it took a while. that one was pinned to version 1.13.2