https://datahubproject.io logo
Join Slack
Powered by
Hi guys, i`m trying to debug issue with ElasticSe...
# troubleshoot
g
Hi guys, i`m trying to debug issue with ElasticSearch. I`m trying to use external ElasticSearch where the user/password were provided to me. ElasticSearch Server is using SSL. When i`m trying to force datahub-gms to use this elasticsearch instances i`m get following errors: When 
SKIP_ELASTICSEARCH_CHECK
is set to false
Copy code
datahub-gms               | 2022/08/31 09:17:40 Problem with request: Get "<https://some.external.elasticsearch.eu:11920>": x509: certificate signed by unknown authority. Sleeping 1s
When
Copy code
ELASTICSEARCH_SSL_PROTOCOL
ELASTICSEARCH_SSL_TRUSTSTORE_FILE/TYPE
ELASTICSEARCH_SSL_KEYSTORE_FILE/TYPE
is undefined and only basic authorization is used
Copy code
datahub-gms               | 2022/08/31 09:20:07 Problem with request: Get "<https://some.external.elasticsearch.eu:11920>": x509: certificate signed by unknown authority. Sleeping 1s
When 
SKIP_ELASTICSEARCH_CHECK
IS SET TO TRUE and rest of 
ELASTICSEARCH_SSL_*
NOT ENABLED During creation of Indexes
Copy code
datahub-gms               | Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is java.lang.RuntimeException: Could not configure system metadata index
datahub-gms               |     at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:224)
datahub-gms               |     at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:117)
datahub-gms               |     at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:311)
datahub-gms               |     ... 42 common frames omitted
datahub-gms               | Caused by: java.lang.RuntimeException: Could not configure system metadata index
datahub-gms               |     at com.linkedin.metadata.systemmetadata.ElasticSearchSystemMetadataService.configure(ElasticSearchSystemMetadataService.java:203)
datahub-gms               |     at com.linkedin.metadata.kafka.hook.UpdateIndicesHook.<init>(UpdateIndicesHook.java:83)
datahub-gms               |     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
datahub-gms               |     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
datahub-gms               |     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
datahub-gms               |     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
datahub-gms               |     at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:211)
datahub-gms               |     ... 44 common frames omitted
datahub-gms               | Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
datahub-gms               |     at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:844)
datahub-gms               |     at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259)
datahub-gms               |     at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
datahub-gms               |     at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
datahub-gms               |     at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598)
datahub-gms               |     at org.elasticsearch.client.IndicesClient.exists(IndicesClient.java:974)
datahub-gms               |     at com.linkedin.metadata.search.elasticsearch.indexbuilder.ESIndexBuilder.buildIndex(ESIndexBuilder.java:51)
datahub-gms               |     at com.linkedin.metadata.systemmetadata.ElasticSearchSystemMetadataService.configure(ElasticSearchSystemMetadataService.java:200)
datahub-gms               |     ... 50 common frames omitted
datahub-gms               | Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
datahub-gms               |     at sun.security.ssl.Alert.createSSLException(Alert.java:131)
datahub-gms               |     at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
datahub-gms               |     at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
datahub-gms               |     at sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
datahub-gms               |     at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
datahub-gms               |     at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
datahub-gms               |     at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
datahub-gms               |     at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
datahub-gms               |     at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
datahub-gms               |     at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:968)
datahub-gms               |     at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:955)
datahub-gms               |     at java.security.AccessController.doPrivileged(Native Method)
datahub-gms               |     at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:902)
datahub-gms               |     at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:285)
datahub-gms               |     at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:345)
datahub-gms               |     at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:523)
datahub-gms               |     at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
datahub-gms               |     at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
datahub-gms               |     at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
datahub-gms               |     at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
datahub-gms               |     at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
datahub-gms               |     at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
datahub-gms               |     at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
datahub-gms               |     at java.lang.Thread.run(Thread.java:748)
With provided own truststore and keystore, the same error as above with PKIX path building failed. With provided certs but with 
SKIP_ELASTICSEARCH_CHECK = FALSE
then the same error as
Copy code
datahub-gms               | 2022/08/31 09:17:40 Problem with request: Get "<https://some.external.elasticsearch.eu:11920>": x509: certificate signed by unknown authority. Sleeping 1s
Any idea what could be wrong ? I were trying to generate truststore and keystore in multiple way and even include cacerts which was provided to me. But all options are failing to me to enable elasticsearch secure communication @blue-megabyte-68048 might be you will be able to look in to it, as you were having similar problem Thanks in advance for hints
And it looks like it was my fault. What works for me: • extracted cacerts from docker image to my local machine • imported rootCA to the extracted cacerts for the server which i would like to connect and • mount as a volume to the specific location ◦ ${HOME}/cacerts:/etc/ssl/certs/java/cacerts After that basic authorization is working fine and i dont need even to provide other parameters like ELASTICSEARCH_SSL_*
s
Hi @great-motherboard-71467 , I am facing the same issue , could you please help me with be more specific on your work around like where you Extracted cacerts and where you imported into ? Thanks in Advance ! PS : i am deploying it on Kubernetes
4 Views
Open in Slack
PreviousNext
Powered by