Hi all, does someone have experience using a kafka...
# troubleshooting
p
Hi all, does someone have experience using a kafka source that's consuming from multiple topics, e.g. using the
topic-pattern
parameter? I'm having one kafka topic per country and consuming those in a Flink job. However, some countries have a lot more messages than others and that leads to the watermark progressing entirely based on the messages from small countries, treating the messages from the other countries all as late. Each country is treated as a separate key in each grouping operation. I thought I could solve that using watermark alignment but it doesn't seem helpful in this case. So what I would ideally need is per-key watermarks, but that would require implementing the windowing mechanism using the datastream api myself, that doesn't seem feasible. Another way would be to run a different Flink job per country, what would increase costs significantly though. I'm wondering if there are any other solutions to this problem?
n
Another way would be to run a different Flink job per country, what would increase costs significantly though.
just curious, are you running an Application cluster or a Session cluster?
p
I'm using application mode, creating a separate FlinkDeployment in a k8s cluster per Flink job.
Running one session cluster for each my current jobs and submitting each country as a separate application to that cluster would be an interesting idea. That would be the same compared to now from a fault tolerance perspective. Also, the infrastructure would the same I suppose. However, I'm slightly concerned regarding the checkpointing costs though since I'd expect to have more files created per checkpoint.
n
yes the idea is probably more of a "this could work" rather than a good solution. Regarding the checkpointing, could I ask what sink you are writing to?
p
The checkpoints are using an s3 filesystem with the presto connector.
n
ah okay, I'm completely unknowledgeable regarding that. We're using a Delta sink and there are some very, very annoying issues regarding memory etc. with it, and when you said this:
I'm slightly concerned regarding the checkpointing costs though since I'd expect to have more files created per checkpoint
I thought maybe you had the same sink and same issues. The heap memory issues we've encountered using Delta have cost me months
p
No, I don't have any experience with that one.