Hi All, I have setup a flink kafka streaming app w...
# troubleshooting
c
Hi All, I have setup a flink kafka streaming app with the job parallelism set equal to number of topic partions, however I am getting into consumer lag issue very often, how to introduce backpressure in flink kafka streaming app to reduce the consumer lag in flink job, any example config would much appreciated.
l
You need to determine the root cause of the consumer lag first, it could be variety of reasons Some first questions to ask would be 1. Is KafkaSource being backpressured due to downstream operators poor performance? 2. Is CPU being maxed out? 3. Does kafka source fetch too little data? From my experience, its not often that kafka/kafka consumer is actually the main bottleneck of the app, though tuning it ofcourse can make a huge difference. For most issues, typically increasing the parallelism of downstream operator may help but it really depends on what your app is doing. GL
c
Thank you @Lukasz Krawiec for your quick response, Here is my app setup on a highlevel, 1. I am running my job from a Java main method directly, I haven't created Flink cluster. 2. I consume events from a Kafka topic which has 11 partitions, and set job parallelism also 11 using below env config. a. env.setParallelism(11); 3. I then query database for each individual event and get some details and attach it to the same event and drop those events which doesn't have any details in database. (here I use stream.flatMap method to transform and filter events) 4. Finally I publish the resultant event to another kafka topic. That's all pretty much my streaming app does.
s
increase the kafka partitions and the parallelism.
c
@Sucheth Shivakumar My primary problem is with the Flink Kafka consumer which consumes the data from those 11 partitions with parallelism set to 11. It's lagging so much so that to process a million events it takes 5-6 hours. Another thing to note here is I am running the job from a EKS cluster and the pod the memory shoots up like very high.
How the increasing partitions and parallelism will reduce the consumer Lag and memory usage, I am not sure, is there any particular reason? Another thing I suspect that I have set up the checkpointing interval to 5 seconds, with this most of the times I see kafka consumer offset commit is failing, so I decided to increase the checkpointing interval to 50 seconds. Is there any best practice of effective checkpointing interval. I am not using any external checkpointing storage, as it's ok to reprocess the events. Because every event is a key value base events, and will be published to a particular partition based on the key.
s
@Chennupati Gopal, that's how it works in the Flink world. If you think your consumer is lagging, you should: 1. Identify the root cause of the lag—there might be something in your business logic causing it. 2. Check if your keys are evenly distributed. Uneven distribution can put a high load on some partitions while leaving others underutilized, leading to bottlenecks. 3. Remember that each task slot is connected to one Kafka partition. Right now, you have 11 slots processing 11 partitions. Increasing the number of partitions and parallel slots can help improve throughput.
c
@Sucheth Shivakumar I totally agree with you, however the kafka topics are existing ones, now I cannot alter the topic and increase the number of partitions, as it has other impact on all other consumers and producer clients in my project. I am also thinking of deploying the job in session mode and create and run separate instances of JobManager, TaskManagers and submit a job. will it be a better approach to solve it? or is there any better way to Identify the root cause, I am kind of running out of options and it's been quite some time debugging it.
Thank you @Lukasz Krawiec Yeah I am using Hikari connection pool in flatMap method to initialise and maintain the connections, is there any standard defined in Flink for Database connectivity, I could not find it any where in the documentation.
l
the async io operator is as standard as you can get I think. Examples are in the docs Main points are you instantiate db connection in open(), and maintain a transient reference to it, which you use in asyncInvoke
c
@Lukasz Krawiec Thank you very much will check it.