Hi All, We are running an Apache Flink application...
# troubleshooting
s
Hi All, We are running an Apache Flink application on production where we are serializing Json messages to Avro. Source (Json messages) : Kafka topic(64 partitions) Sink (Avro messages) : Kafka topic(64 partitions) Flink version: 1.7.1 We are running the job on 84 parallelism. The issue is that we are observing High backpressure on the Kafka Source vertex because of which Kafka lag is increasing. Initially, we suspected Kafka Sink for the backpressure on the source. So we tried with the Null Discarding Sink but still the backpressure persisted. So now we suspect Avro serialization since we are not doing any other processing in the job apart from some filter and map operators. Can Avro serialization cause backpressure? Or Are we missing something here? Need help from the community on this. If Avro serialization is the problem then How to mitigate it?