Hey team. We are using Confluent cloud. In this setup, our CRM system produces raw data on a Kafka topic. Then a Flink job (running in AWS EMR) transforms this raw data into meaningful data and stores on another topic. Problem statement: This Flink jobs gives errors beyond a certain number of events ( for reference - we have provisioned for 100k events but the Flink job fizzles beyond 5-10k events, stop and resumes after 10-20 mins). On the confluent side, there are no apparent errors as it's a managed service and no throttling has been configured. Summarized Error description: Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings". Current keepalive time (before throttling): 20000ms

Some points : 1. Check if CPU usage is high on your confluent cluster 2. Monitor average batch size and request rate of the producer to understand if you are making too many calls to cluster. 3. If step 2 gives you a hint that too many requests are causing problem (leading to high cpu usage), tune batch.size and linger.ms of the kafka producer. eg: Increase default batch.size to 32768(32Kb). default value is 16kb Add some delay while producing. You can set linger.ms=10ms. Please tune based on your cluster resources and data rate.

hey thanks Sumit. Some thoughts on your points: 1. This is confluent cloud so this is taken care of by the service. 2. Will check this one. But from what I know, the flink team has sized for 100K requests. Will ask them to re verify all parameters. 3. Will check this one too.