Been studying the Flink internals, and still feeling a bit epistemically shaky around choosing steps forward. The platform we’re migrating away from is Kafka Streams. Its biggest pain point is the cost of rebuilding all internal state from source topics following a release. I’m worried that if I just use a KafkaSource, I’m going to have a similar problem – just deserializing and processing full rows of records that may or may not be relevant to the present computation. I see two angles. Flink Table Store, with a hybrid source, but it looks like I’d need a custom offset tracking mechanism in the bulk load… or just dump my Kafka topics to a data source with a Flink CDC connector. I want to take full advantage of the work that’s gone into the Table API over the past 3+ years, and it looks like different connectors play nicer with Table API optimization rules? That’s where my head is, anyway. Anybody have relevant color or pointers?

We’re in a similar situation, but our Spark jobs are batch jobs, and we’re writing some new streaming jobs with Flink. I’m writing the code in Python, and it’s allmost all SQL. I find that it’s easier to communicate with data scientists if it’s SQL.