lemon-cat-72045
10/20/2022, 4:44 PMdatahub-kafka-setup
job when we are connecting to our own Kafka?
• What does datahub-kafka-setup
do exactly? Since when I check the helm template it seems that all components have their own Kafka configurations.
• I found that some of the deployment.yaml
in the subchart template uses KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }}
but some uses SPRING_KAFKA_PROPERTIES_{{ $configName | replace "." "_" | upper }}
. May I know the difference between this to configuration settings?lemon-cat-72045
10/20/2022, 4:46 PMcredentialsAndCertsSecrets:
name: datahub-certs
path: /mnt/datahub-kafka/certs
secureEnv:
ssl.truststore.password: truststore.password
kafkastore.ssl.truststore.password: truststore.password
springKafkaConfigurationOverrides:
security.protocol: SASL_SSL
sasl.mechanism: SCRAM-SHA-256
ssl.ca.location: /mnt/datahub-kafka/certs/ca.crt
ssl.truststore.location: /mnt/datahub-kafka/certs/truststore.jks
kafkastore.ssl.truststore.location: /mnt/datahub-kafka/certs/truststore.jks
sasl.jaas.config: "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"testUser\" password=\"testPassword\";"
lemon-cat-72045
10/20/2022, 4:49 PMdatahub-kafka-setup
job
%6|1666283192.838|FAIL|rdkafka#consumer-1| [thrd:<http://kafka-1.preprod.testing.im:19092/bootstrap|kafka-1.preprod.testing.im:19092/bootstrap>]: <http://kafka-1.preprod.testing.im:19092/bootstrap|kafka-1.preprod.testing.im:19092/bootstrap>: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 4ms in state APIVERSION_QUERY, 3 identical error(s) suppressed)
lemon-cat-72045
10/20/2022, 4:53 PMmae-consumer
11:15:44.795 [R2 Nio Event Loop-1-1] WARN c.l.r.t.h.c.c.ChannelPoolLifecycle - Failed to create channel, remote=datahub-datahub-gms/10.67.3.210:8080
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: datahub-datahub-gms/10.67.3.210:8080
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:829)
11:15:44.805 [pool-7-thread-1] ERROR c.d.m.ingestion.IngestionScheduler - Failed to retrieve ingestion sources! Skipping updating schedule cache until next refresh. start: 0, count: 30
com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-datahub-gms:8080/entities>
at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)
at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponse(ResponseFutureImpl.java:94)
at com.linkedin.common.client.BaseClient.sendClientRequest(BaseClient.java:36)
at com.linkedin.entity.client.RestliEntityClient.list(RestliEntityClient.java:361)
at com.datahub.metadata.ingestion.IngestionScheduler$BatchRefreshSchedulesRunnable.run(IngestionScheduler.java:216)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-datahub-gms:8080/entities>
at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:67)
at com.linkedin.r2.transport.http.client.rest.ExecutionCallback.lambda$onResponse$0(ExecutionCallback.java:64)
... 3 common frames omitted
Caused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: datahub-datahub-gms/10.67.3.210:8080
at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.onError(ChannelPoolLifecycle.java:142)
at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.lambda$create$0(ChannelPoolLifecycle.java:97)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
... 1 common frames omitted
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: datahub-datahub-gms/10.67.3.210:8080
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:829)
I'm guessing this is caused by enabling standalone consumers and the pods's network needs to be configured?broad-alarm-60780
10/20/2022, 8:51 PMbroad-alarm-60780
10/20/2022, 8:53 PMbroad-alarm-60780
10/20/2022, 8:54 PMlemon-cat-72045
10/21/2022, 6:52 AMlemon-cat-72045
10/21/2022, 6:59 AMkafka:
bootstrap:
server: "<http://kafka-1.preprod.test.im:19092,kafka-2.preprod.test.im:19092,kafka-3.preprod.test.im:19092|kafka-1.preprod.test.im:19092,kafka-2.preprod.test.im:19092,kafka-3.preprod.test.im:19092>"
schemaregistry:
url: "<http://kafka-1.preprod.test.im:8081>,<http://kafka-2.preprod.test.im:8081>,<http://kafka-3.preprod.test.im:8081>"
credentialsAndCertsSecrets:
name: datahub-certs
path: /mnt/datahub-kafka/certs
secureEnv:
ssl.truststore.password: truststore.password
springKafkaConfigurationOverrides:
security.protocol: SASL_SSL
sasl.mechanism: SCRAM-SHA-256
ssl.truststore.location: /mnt/datahub-kafka/certs/truststore.jks
ssl.ca.location: /mnt/datahub-kafka/certs/ca.crt
sasl.jaas.config: "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"testuser\" password=\"testPass\";"
But the datahub-kafka-setup-job
is unable to run and failing with the following errors
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Am I missing something in the SSL configuration? Thanksbroad-alarm-60780
10/21/2022, 9:01 PMbroad-alarm-60780
10/21/2022, 9:01 PMsudo "$JAVA_HOME/bin/keytool" -import -keystore "$JAVA_HOME/jre/lib/security/cacerts" -storepass changeit -noprompt -file YOUR-ROOT.cer -alias ANYNAME-YOU-WANT
broad-alarm-60780
10/21/2022, 9:02 PMlemon-cat-72045
10/24/2022, 3:27 AM