<@U04KC3KFW77> not uncommon to encounter issues wh...
# troubleshooting
d
@Dheeraj Panangat not uncommon to encounter issues where Apache Flink applications using Kafka as a source creates an excessive number of connections to the Kafka brokers. You can check Parallelism vs Num Brokers. If you flink job has a high parallelism for the Kafka source then each instance may try to establish it’s own connection to the Kafka brokers. If more parallel source tasks than brokers it could lead to inflated number of connections.
d
Thanks @D. Draco O'Brien for the response. We have a parallelism of only 40, but the connection created by the job are 992, which is way higher. Would you know if it could have anything to do with the broker or Flink creates lot more ?
d
Well maybe you need to ensure that all parallel tasks in your Flink job are using the same consumer Group ID? because if each is using a different id it could lead to multiplication of connections.
On the broker side you can check max.connections.per.ip and connections.max.idle.ms that they are not too restrictive
also Flink rescaling without closing connections properly could cause excessive number of connections
d
Thanks @D. Draco O'Brien. • Yes, they all are using same Group ID • The Flink jobs are working fine and getting the connections they need, it is our Kafka Broker that is failing because of too many connections. • We have not enabled autoscaling
d
Do you close connections on rescaling?
what are your settings for max.connections.per.ip and connections.max.idle.ms? you could try increasing these I think
d
Hi @D. Draco O'Brien, we have kept that as default. Would you know if Producers would create more connections ?
d
Well you should have debug logging enabled and maybe you can see it with netstat as well?
Copy code
netstat -anp | grep :80 | grep ESTABLISHED | wc -l
but replacing with ports or port range
range of ports
Copy code
netstat -nau '( sport >= 1000 and sport <= 10000 )'
d
Hi @D. Draco O'Brien, We did check the connections using Flink Kafka Metrics We have 6 brokers and the topics have 20 partitions and are equally distributed across brokers. Our Flink parallelism is 20 to match the partitions, but we end up having 20 parallelism for sink which means kafka producers are 20 and each connect to all brokers (6) making 120 connection in total. For some application I see (which is correct)
Copy code
{
  "id": "Sink__Unnamed.KafkaProducer.connection-count",
  "min": 6,
  "max": 6,
  "avg": 6,
  "sum": 120
}
For some I see (why does it not connect to all brokers)
Copy code
{
  "id": "Sink__Unnamed.KafkaProducer.connection-count",
  "min": 1,
  "max": 1,
  "avg": 1,
  "sum": 20
}
For Kafka consumer 1 connection per partition, making 20 connections with 20 parallelism, But in Flink Kafka Metrics it shows 52 or for some I see sum 40
Copy code
{
  "id": "Source__Custom_Source.KafkaConsumer.connection-count",
  "min": 2,
  "max": 3,
  "avg": 2.6,
  "sum": 52
}
Any thoughts on these discrepancies ? Is my understanding correct ? FYI: @Kiran Aswath @Aakshi Nagpal @Mahesh Gupta @Aman Vaishya @Mrinal Ekka
d
when you observe on connection per producer (sum of 20 connections). It does suggest that these producers might be configured differently or encountering issues connecting to all brokers. It could be that the producer’s bootstrap.servers list is not correctly set to include all brokers. If dns names are used maybe the dns entries don’t work from these particular producers or they are missing a local host file. Or network restrictions might prevent these producers from reach all brokers.
Kafka consumers also establish connections for coordination which can explain why some have more connections as well.
Sometimes consumers are fetching metadata about topic for instance which can lead to addition connections as well.
there is also consumer group management that takes place and depending on client used by flink(and maybe version) and it’s configuration additional connections could be made to manage offsets in Kafka.
Make sure all consumers/producers are configured to use ALL the brokers that are available as boostrap.servers. Maybe test connectivity to them as well.
Check network connectivity to these brokers and how it might very depending on the flink application
And of course you should probably enhance networking monitoring so that you can see what connections are being established and what they are for. I would actually probably start here with enhanced network monitoring.
b
you can also do a stacktrace to an an idea where are these connection coming from