What is likely causing this issue, causing my Flin...
# troubleshooting
a
What is likely causing this issue, causing my Flink jobs to crash?
Copy code
2024-06-14 14:34:57,752 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_e7a7ee01-8dbc-44f5-8265-cc9c14522fdc-enumerator-admin-client] Node -1 disconnected.
2024-06-14 14:34:57,752 WARN  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_e7a7ee01-8dbc-44f5-8265-cc9c14522fdc-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Broker may not be available.
If I get a terminal session on my jobmanager docker container and install nslookup and ping, I can do both successfully for dev-app.tec.truplatform.io. SSL is enabled on Kafka.
d
job manager restarting maybe?
a
Hi David. No, it doesn't. The job manager logs just get flooded with messages such as these:
Copy code
2024-06-14 15:48:22,754 WARN  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_a7c522bc-78d2-4b97-ba99-2ec5c8202c3a-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
2024-06-14 15:48:22,955 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_b9649564-613f-4336-bb97-2355ed76432d-enumerator-admin-client] Node -1 disconnected.
2024-06-14 15:48:22,955 WARN  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_b9649564-613f-4336-bb97-2355ed76432d-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
2024-06-14 15:48:23,017 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_bbb63520-be66-429c-8049-c27904dafed7-enumerator-admin-client] Node -1 disconnected.
2024-06-14 15:48:23,017 WARN  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_bbb63520-be66-429c-8049-c27904dafed7-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
2024-06-14 15:48:23,471 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_ee1556db-97bd-4fde-b282-07d47aee5d5e-enumerator-admin-client] Node -1 disconnected.
2024-06-14 15:48:23,471 WARN  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=tru_platform_events_ee1556db-97bd-4fde-b282-07d47aee5d5e-enumerator-admin-client] Connection to node -1 (dev-app.tec.truplatform.io/192.168.15.196:9092) could not be established. Node may not be available.
The jobs stay alive for quite a while because of the restart policies, but eventually die. Looks to me to be related to the
import org.apache.flink.connector.kafka.sink.KafkaSink;
class. Perhaps something missing from the KafkaProducerConfig?
Copy code
KafkaSink<String> kafkaSink;
    try {
      kafkaSink =
          KafkaSink.<String>builder()
              .setBootstrapServers(KafkaConnector.getKafkaServerUrl())
              .setKafkaProducerConfig(clonedProperties)
              .setDeliveryGuarantee(deliveryGuarantee)
              .setTransactionalIdPrefix(uniqueProducerId)
              .setRecordSerializer(recordSerializer)
              .build();
      return kafkaSink;
    } catch (Exception e) {
      throw new ProducerCreationException("Failed to create producer for topic: " + topic, e);
    }
d
Very strange, I don't have much experience with the Kafka connector.
k
install nslookup and ping, I can do both successfully for dev-app.tec.truplatform.io.
Since basic connectivity appears to be established, are the Kafka ports open? I'd suggest also trying with
kcat
. https://github.com/edenhill/kcat
πŸ‘ 1