Hi I am looking at how I can solve a problem using...
# troubleshooting
a
Hi I am looking at how I can solve a problem using Flink.. I have two topics in Kafka P1 and P2. I want Flink to consume from both P1 and P2 but backpressure and pause consuming from P2 while P1 has messages. When P1 messages are over P2 can resume and if P1 never gets completed I dont mind if Kafka drops old messages in P2. I could probably do it using custom source function in Kafka but I dont like the idea of not using native Flink leading to edge cases during parallel processing of partitions. What is the best way to do priority based message consumption in Flink?
p
My two cents - 1- First try with async sink, and if this can help you to reduce lag in P1 kafka topic. 2- Second- Use custom deserialiser and introduce isFilter flag and kafka topic name in the pojo, and in your operator when you detect a back pressure check the current record topic name and if it is secondary one set the filter to true and this way you can filter out the records from downstream pipe line. The only issue is, it is not pause but complete drop.
I just started using flink, so I am not sure wheather this is achievable completely. But if I will come with such scenario I will definately try this way as a general solution