Hi Team, I am new to apache flink and trying to use flink with spring boot java. I want to inject some of my business specific classes in custom implementation of process function. Can anyone suggest me the way forward.

TL;DR IMHO its not possible. For e.g. if u want to use the DI of Spring Boot. And the invest integrating Flink into Spring would not be worth it. Use Flink directly IMHO

Thanks @Akhlaq Malik for quick reply. Is there any workaround around this ? Like using RichFunction or Dagger

@Abhishek Gupta @Akhlaq Malik actually it is possible to use Spring's DI in your Flink job without significant cost. How I did it is that I've created an uber jar with spring only dependencies and I've puted it under the flink/lib folder on TM and JM nodes. that uber jar contains one custom class called "ContextRegistry" In my job pom.xml I;ve added as a dependency that uber flink job. like so:

<dependency>
  <groupId>com.getindata</groupId>
  <artifactId>flink-spring</artifactId>
  <version>0.1.0-SNAPSHOT</version>
  <scope>provided</scope>
</dependency>

And I did this: https://github.com/getindata/flink-http-connector/commit/6049c87c7d9319dbe242792601e00d24b07ef838 We finally step back from using Spring to use Flink;s "Factory" mechanism for Implementation injection; however I do think that this Spring PoC can be beneficial. I'm planning to prepare a Tech Blog post for this one in upcoming weeks, so stay tuned.

thanks Krzysztof for the links and the integration work that u have done. Feels still little bit complicated to integrate Spring into Flink or other way around like putting jars into TM/JB to be working and all the CI/CD stuff that would one need to do or maintain ... The question for what benefits :) IMHO If Spring is required with all his ecosystem and one has to do stream processing with Kafka as source/sink I'd go with Kafka Streams.

sure it all depends on the needs. Spring-like apps are widely common and familiar. If your company and developers are used to Spring apps, they most likely would like to keep that pattern while implementing business logic that will be run on Flink and at the same time, they might have microservices that would run the same logic/libs. In fact Flink can abstract out from that logic, in a way that your Business Java Team would not know that this will be run on Flink. So in general, I would say that the benefit is rather in using the Dependency injection pattern and if you would like to use Spring to "run" this pattern then here is how you can do it. The benefit is that there are more people that know how to write Spring apps than Flink jobs and it's not only about the API. Being able to port already existing business libs that are using Spring for DI to Flink is a potentially cool thing.

totally agree 🙂 will have look later on the project you have shared. Such a integration would potentially also open Flink as an option to a giant Spring community

btw, we have followed Dependency Injection in our HTTP connector [1] . We wanted to give users the way to implement their own formats, handlers etc for SQL/Table API but we stick with the Flink Factory pattern [2][3] which generally speaking provides dependency injection, in some way 🙂 [1] https://github.com/getindata/flink-http-connector [2] https://github.com/getindata/flink-http-connector#custom-rest-query [3] https://github.com/getindata/flink-http-connector#custom-requestresponse-callback

Thanks a lot @Krzysztof Chmielewski @Akhlaq Malik for your inputs 🙂