Hey all, I’m looking into using Flink SQL for CDC and creating real time analytics databases. I can use the SQL cli, and that works fine for a demo. Now I’m looking into a more mature setup. I’d like a git repo containing the SQL queries, and then a CI/CD pipeline updating / deploying those to my Flink cluster, probably running using the new K8s operator. Before (badly) reinventing stuff, any pointers on where to start?

We implemented the entire thing: • We have a custom helm chart that deploys a FlinkDeployment with the

SqlRunner.jar

from the

examples

folder • We load the queries from the Helm’s values file The main challenge was to setup the image with all the dependencies, since we output to S3 through Hive, and setting Apache dependencies is a b*tch That being said, other than very specific very lean pipelines - we use the DataStream API.

Ah cool. Yeah I figured creating the image is challenging.

The main pain point is resuming from checkpoint. Since your application will probably be stateless - it shouldn't be a problem. Other concerns, like logging, monitoring and scaling are a breeze (as long as you remember that scaling requires a job restart).

Cool, this has helped me a lot. Thanks!