Peter Klauke
05/30/2024, 8:34 AMtopic-pattern parameter?
I'm having one kafka topic per country and consuming those in a Flink job. However, some countries have a lot more messages than others and that leads to the watermark progressing entirely based on the messages from small countries, treating the messages from the other countries all as late. Each country is treated as a separate key in each grouping operation.
I thought I could solve that using watermark alignment but it doesn't seem helpful in this case. So what I would ideally need is per-key watermarks, but that would require implementing the windowing mechanism using the datastream api myself, that doesn't seem feasible. Another way would be to run a different Flink job per country, what would increase costs significantly though.
I'm wondering if there are any other solutions to this problem?Nicholas Erasmus
05/30/2024, 10:58 AMAnother way would be to run a different Flink job per country, what would increase costs significantly though.just curious, are you running an Application cluster or a Session cluster?
Peter Klauke
05/30/2024, 12:09 PMPeter Klauke
05/30/2024, 12:47 PMNicholas Erasmus
05/30/2024, 12:49 PMPeter Klauke
05/30/2024, 12:52 PMNicholas Erasmus
05/30/2024, 12:55 PMI'm slightly concerned regarding the checkpointing costs though since I'd expect to have more files created per checkpointI thought maybe you had the same sink and same issues. The heap memory issues we've encountered using Delta have cost me months
Peter Klauke
05/30/2024, 1:03 PM