I’m facing an issue with Kafka consumption in my ...
# troubleshooting
a
I’m facing an issue with Kafka consumption in my Flink application. The Kafka dashboard shows that the fetch rate of one specific broker, which holds the leader partitions of the Kafka topic, has suddenly dropped to zero, even though the data received by Kafka remains steady. On the Flink side, most of the tasks appear busy, except for three that are idle. There is no backpressure reported in the Flink application. What could cause the fetch rate from this broker to drop to zero while data is still being produced, and some Flink tasks remain idle? Could this be related to partition assignment, broker issues, or other factors? Any suggestions on how to debug or resolve this would be greatly appreciated.
j
Do you happen to have a parallelism higher than the number of topic-partitions you are consuming from? It could be an issue with Idleness : https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/kafka/#idleness
a
No, I configured it with the same number of Kafka partitions, and it starts off working well. However, after some time, I notice that the fetch rate in the dashboard of one specific broker drops to zero. In Flink, the sub-tasks that consume from this broker become idle. When I restart the broker, the leader partitions of this topic get redistributed, and then Flink can start fetching data properly again—until the issue reoccurs.
I doubt that the problem in kafka not in flink , because there is no back-pressure
j
What you are describing (three task not being busy and no backpressure) still points me to an idleness issue though… I would try to set something like
Copy code
table.exec.source.idle-timeout: 100ms
and see if that unblocks progress
a
Yes it is idle from flink Pov , but in kafka , the lag in consumer offset started to increase in these partitions , because the data is being produced at these partitions
j
Yeah if the consumers don’t consume anything the lag will increase
a
Yes , but It works good if I redistribute the leader of partitions , but the issue occurs again after while
j
Im thinking something like skewed data and some partitions don’t get new data ?
Im not sure if redistributing the leader would re-allocate the partition to a new consumer which could explain what you’re seeing
a
The problem here is related to kafka more than flink , flink works good
j
Ok if you think so, then I can’t say 🤷
a
The suddenly downing in fetch of the broker is kafka issue not a flink issue because when the leader of the partitions goes to another brokers it works good
j
If it’s a bad broker for example you will see Kafka errors in flink logs
a
Can you please explain more this ? Im not sure if redistributing the leader would re-allocate the partition to a new consumer which could explain what you’re seeing
j
if it was me i’d check the watermarks tab for source task in flink ui
a
Yes , i checked it and the watermark of one of idle subtask consumer is falling behind(1 hour ago) , after redistrbute the leader of the partitions the watermark be 4 mins ago
j
Is there a way you can check if the data is not skewed ? ie; only being produced to a subset of the partitions ?
a
I recently noticed a memory issue(cassandra and flink computing for memory) on the Flink TaskManager node. After resolving it, Flink started consuming data from Kafka properly. However, I still have two unresolved questions: a. Why didn’t this issue show up as backpressure in the Flink UI? b. Why did restarting the Kafka broker with low throughput cause Flink to consume data properly for a while before the issue reappeared?