what's the thoughts on having a realtime flink pipeline that depends on a DB for looking up historical data? we have a pipeline where, say we have a login event we want to re-process some prior data and enrich it with a new user association. i'm guessing in flink i could have a DB connector and keep say N days of data in there for lookups to re-process. the caveat is that it's technically not an event-time processing pipeline anymore and we're producing duplicate event streams which have to be deduplicated/handled down stream (i.e. it becomes almost a batched process). what are the thoughts on this?

Does the historical data change?

hey @Marco Villalobos, nah the data will be the same and we're using the data stream API for this at the moment

I see. This type of problem is actually more easily solved with the SQL API and the Look Up join as specified in https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/ This approach even supports caching the data so you don't have to look it up after that. However, since you using the data-stream API then there are two approaches that would work. The first simple approach is to use a

KeyedProcessFunction

that is keyed by the unique key of whatever data that you're retrieving. Use keyed state to store the data retrieved by key. If the keyed state is empty perform your database query with JDBC (you create a connection do the query and then disconnect) then store the data in keyed state. This is using keyed state as a cache. The second more complicated approach, and the better solution in my opinion, is for your stream element to support multiple types of data, control data and stream data to control the stream job. It is briefly touched in this article: https://www.mux.com/blog/using-control-streams-to-manage-apache-flink-applications The way you do that is instead of looking up the data with JDBC, you query your historical data and put it in your kafka source as control data. Key the data by its key, and store it in keyed state. When your data stream elements are processed, you can key the same key and then associate the data. In my code, I don't use a different topic, I just put both control data and stream data on the same kafka topic, and I encode it with Avro. I hope you understand what I mean.