Hi Fellas. I have been looking into Flink for some time now and have concluded not to use it for our project. The reason I was looking into it is because we want a tool that can stream from multiple sources and produce and sink a combined message to kafka. The reason I decided that its not ment for our use case is that the messages which should be joined from the different topics can be updated and they are neither created nor updated near eachother. So cant join using a small window. Making the state size increase indefenetly. Another reason is that in most usecases if joining multiple topics then any update to either of the three topics should result in new messages for all messages using those data. So if a car model used a colour and the combined message is carAndColour then if the colour name was updated. Then all messages with carAndColour using that colour should be resent. This would also require all messages to be kept in state. And I read that flink is ment for "fire and forget" not going back to process old messages. This was the short version of my conclusion. If youd like to read the full one send me a dm and Ill share it. And if you have comments both if you agree or not I would really like to hear it. Thx you. Ole B.

And I read that flink is ment for "fire and forget" not going back to process old messages.

This is definitely not true. When you're performing a regular streaming join in Flink, it needs to keep all the joined result in state because it might need to retract that value in the future, because of an update on the earlier joined result. I believe that's exactly what you're describing here

Ok, Ill take a look at the dynamic tables. But my ttl will have to be forever. Unless an item is marked as deleted. In my conclusion I have also written that the jobs are unable to share state. So if we wrote another job that used many of the same topics they would be dublicated in the state.

Well I could still come to the same conclusion but I don’t have enough info. What is your latency requirement, expected load, use case etc. Also what is acceptable in terms of updating earlier results (it implies infinite, since else the state size wouldn’t be infinite)

Latency requirement. We dont have a requirement. But lets say one hour. Expected Load. The biggest topic we need to join has 13 million messages. It gets about 15000 messages per day. The second largest has 1 million messages. And gets 3000 messages per day. I have to update all earlier results. This usecase Im mentioning is to gather data from 6 different topics to describe an item. So a user can search for an item number and get more information about it. The reason we want to create this joined product is because we need it in many services. So to avoid doing the same logic in all services. We want to create the product and make it available. If we decide not to use flink. We would code it ourself in a spring boot service with a postgres database behind it. Still consume from kafka but then join it in the database and push it back to kafka. This usecase is one job. But if we start using flink, more will come, that will sometimes use the same source topics.

@Ole Bille thanks for sharing your reasoning, it's interesting. May I ask what you've decided to use instead of Flink?

You might want to consider using clickhouse with materialized views instead of postgres. That is an option that I have used to replace some unwieldy flink jobs.

What you describes reminds me a bit the talk from Yaroslav: Flink could be relatively easy scaled up to TB of state in modern cloud providers.

Yes we do massive joins at hundreds of thousands of messages per second. You need state no matter which method you use.

*Reply to @rmoff "*thanks for sharing your reasoning, it's interesting. May I ask what you've decided to use instead of Flink" The plan is to create a new spring boot service that will listen to the 6 topics and save those to individual tables. When we create or update we will also save a task with the updated rows id. And when the data is saved we will trigger a scheduled task to send the messages to the new topic. To create the joined topic we will select with a join on all 6 tables and push those messages to kafka. It is our current plan. But I havent looked into clickhouse mentioned by @Stephen Pendleton and the video posted by @Alex Nitavsky