https://flink.apache.org/ logo
Join Slack
Powered by
Hi, i have trouble with communicate with kafka con...
# troubleshooting
a
Hi, i have trouble with communicate with kafka configured with ssl. producer has the following config:
Copy code
security.protocol = SSL
        ssl.truststore.type = JKS
        ssl.truststore.location = /store/truststore.jks
        ssl.keystore.type = JKS
        ssl.keystore.location = /store/my-keystore.jks
        ssl.keystore.password=keystore_password
        ssl.truststore.password=trustore_password
I'm getting SSL handshake error in kafka broker side, and this error in flink job manager:
Copy code
java.io.IOException: Connection to ******:9093 (id: -1 rack: null) failed.
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70) ~[validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.awaitNodeReady(Sender.java:526) ~[validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:447) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
Copy code
java.io.EOFException: null
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:120) ~[validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) ~[validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:73) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.awaitNodeReady(Sender.java:526) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.maybeSendAndPollTransactionalRequest(Sender.java:447) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:316) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [validator_2.0.0-RC2-48-d6f38ff4.jar:2.0.0-RC2-48-d6f38ff4]
Important: i have another microservice that works fine with kafka. any ideas?
r
Are you sure the security protocol is SSL and not SASL_SSL?
l
Hey Reme, yes, the protocol is SSL. I am not sure flink is using the mentioned SSL properties, maybe we did not pass them correctly(?)
r
Are you using 
.setProperties()
to set the security protocols? Mind sharing your code?
l
I need to check about sharing the code, but yes, 
.setProperties()
is used:
Copy code
def build: KafkaSource[A] = KafkaSource
    ...
    .setProperties(readKafkaConfigs())
    .build
the 
readKafkaConfigs
function fetches the ssl configuration
We noticed the following today: 1. the issue happens only on the job manager 2. the job manager initializes two kafka clients 
ProducerConfig
and 
AdminClientConfig
in 
ProducerConfig
we don’t see our ssl configs:
Copy code
security.protocol = PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        <http://socket.connection.setup.timeout.max.ms|socket.connection.setup.timeout.max.ms> = 30000
        <http://socket.connection.setup.timeout.ms|socket.connection.setup.timeout.ms> = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
but in the 
AdminClientConfig
we do see everything as expected:
Copy code
security.protocol = SSL
        security.providers = null
        send.buffer.bytes = 131072
        <http://socket.connection.setup.timeout.max.ms|socket.connection.setup.timeout.max.ms> = 30000
        <http://socket.connection.setup.timeout.ms|socket.connection.setup.timeout.ms> = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = /store/my-keystore.jks
        ssl.keystore.password = [hidden]
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = /store/truststore.jks
        ssl.truststore.password = [hidden]
        ssl.truststore.type = JKS
any idea why is this happening?
we found the issue. Our dev team are initializing two clients but passed the ssl configs only to one of them 🙂
4 Views
Open in Slack
PreviousNext
Powered by