Hi, I was wondering how can I optimize the distrib...
# troubleshootinga
Adesh Dsilva
01/27/2023, 9:31 PMHi, I was wondering how can I optimize the distribution of topic partitions across my kafka consumers
I have 5 kafka topics, with 4, 4, 4, 2, 2 partitions (each topic respectively) which comes to 16 partitions.
I started flink with parallelism of 8 with 2 slots per TM. Attached is the screenshot where you can see one sub task is idle.
How can I ensure all my sub tasks are assigned 2 partitions each so that all of them are consuming messages? (Want to avoid any rebalance or shuffle if possible)
Adesh Dsilva
01/30/2023, 9:48 AMDoes anyone have any recommendations?
I can think of two solutions here...what I can do is create 2 consumers, one consumer will have 4 parallelism taking input from topics with 4 partitions and another with 2 parallelism taking input from topics with 2 partitions. Then either have them as separate pipelines or do an union not sure which would be better.
The other solution is to use rebalance after consumer reads so that all tasks still do the work other than input processing. Is there any other better solution? I know exactly how many records I get from my input per hour so I want it to be as optimal as possible.
s
Sandeep Kongathi
01/30/2023, 10:18 AMnot sure if it help, I read this article fre weeks back - https://medium.com/@sanr_71172/dealing-with-data-skew-in-flink-b7e4c82c35ef
a
Adesh Dsilva
01/30/2023, 10:45 AMThanks will give it a read