Alex Bryant
06/14/2024, 3:24 PM2024-06-14 14:34:57,752 INFO org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_e7a7ee01-8dbc-44f5-8265-cc9c14522fdc-enumerator-admin-client] Node -1 disconnected.
2024-06-14 14:34:57,752 WARN org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_e7a7ee01-8dbc-44f5-8265-cc9c14522fdc-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Broker may not be available.
If I get a terminal session on my jobmanager docker container and install nslookup and ping, I can do both successfully for dev-app.tec.truplatform.io. SSL is enabled on Kafka.David Bryson
06/14/2024, 4:11 PMAlex Bryant
06/14/2024, 4:21 PM2024-06-14 15:48:22,754 WARN org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_a7c522bc-78d2-4b97-ba99-2ec5c8202c3a-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
2024-06-14 15:48:22,955 INFO org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_b9649564-613f-4336-bb97-2355ed76432d-enumerator-admin-client] Node -1 disconnected.
2024-06-14 15:48:22,955 WARN org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_b9649564-613f-4336-bb97-2355ed76432d-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
2024-06-14 15:48:23,017 INFO org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_bbb63520-be66-429c-8049-c27904dafed7-enumerator-admin-client] Node -1 disconnected.
2024-06-14 15:48:23,017 WARN org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_bbb63520-be66-429c-8049-c27904dafed7-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
2024-06-14 15:48:23,471 INFO org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_ee1556db-97bd-4fde-b282-07d47aee5d5e-enumerator-admin-client] Node -1 disconnected.
2024-06-14 15:48:23,471 WARN org.apache.kafka.clients.NetworkClient [] - [AdminClient clientId=tru_platform_events_ee1556db-97bd-4fde-b282-07d47aee5d5e-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
The jobs stay alive for quite a while because of the restart policies, but eventually die.
Looks to me to be related to the import org.apache.flink.connector.kafka.sink.KafkaSink; class.
Perhaps something missing from the KafkaProducerConfig?
KafkaSink<String> kafkaSink;
try {
kafkaSink =
KafkaSink.<String>builder()
.setBootstrapServers(KafkaConnector.getKafkaServerUrl())
.setKafkaProducerConfig(clonedProperties)
.setDeliveryGuarantee(deliveryGuarantee)
.setTransactionalIdPrefix(uniqueProducerId)
.setRecordSerializer(recordSerializer)
.build();
return kafkaSink;
} catch (Exception e) {
throw new ProducerCreationException("Failed to create producer for topic: " + topic, e);
}David Bryson
06/14/2024, 4:41 PMKirill Ternovsky
06/14/2024, 5:57 PMinstall nslookup and ping, I can do both successfully for dev-app.tec.truplatform.io.Since basic connectivity appears to be established, are the Kafka ports open? I'd suggest also trying with
kcat. https://github.com/edenhill/kcat