Flink support batch mode or streaming mode. Is there any support for "hybrid" mode? When working with debezium connectors, most jobs will first have a snapshot stage, and then a streaming stage. Currently, the only option is to run the entire job as a streaming job. This means even the snapshot stage runs in streaming mode. This leads to a very heavy performance penalty. For example, consider calculating rank. The sink tables will unnecessarily be written with all the intermediate values until rank=1 is found. This leads to a lot of dead tupes in the sink tables, which slows down the job a lot.. So my question is - is there anyway to start a job in batch mode, and then once the debezium snapshot is finished, run the same job in streaming mode

IIUC, you want https://cwiki.apache.org/confluence/display/FLINK/FLIP-327%3A+Support+switching+from+[…]ream+mode+to+improve+throughput+when+processing+backlog+data It's WIP now.

Do you know if its currently planned for version 2.0 or will it miss that release? I don’t see it in the confluence doc for 2.0 nor the doc for 1.20 but in the jira ticket itself its tagged as 1.20.

cc @Xuannan Su

Currently, the infrastructure is finished, i.e., the job can propagate the RecordAttributes with backlog status. The source implementation will have to send the backlog status and the downstream operator can be customized to utilize the backlog status to change the behavior. The MySQL CDC source will be able to send backlog status in the coming release for Flink 1.20. The end goal of FLIP-327 is to adjust the behavior of the operator based on the backlog status without changing the current implementation of the operator, which requires more work than we expected so it is possible that we may miss the release of 2.0.

Thanks for the update! My team within my company is anticipating this feature highly. We have been considering building our own system to stop a job after running in batch with a file source to process backlog and then restarting in streaming swapping the file source for a kafka source, but we’re not sure how easily this could be accomplished/whether the savepoint of the batch job will work when switched over.

thanks for the replies! This feature will be a true killer feature.. it will take flink to the next level