https://datahubproject.io logo
Join Slack
Powered by
@here Can anyone from Datahub team help me this MR...
# ingestion
g
@mammoth-bear-12532 / @big-carpet-38439 This is blocker for us if anyone from datahub can help us in review and merge of this MR please!! https://github.com/datahub-project/datahub/pull/5968 related to issue https://github.com/datahub-project/datahub/issues/5872
m
@great-branch-515 your PR seems to just change the declaration of the dependency on the jar. I don't see how this will ensure that you find the jar on the classpath in the server.
Also, are there no code changes needed?
In that case it might be simplest to just drop in this jar as a runtime jar into your deployment?
g
We just need dependancy Once Jar is available, I can override global variables in K8s configuration like this
Copy code
global:
  graph_service_impl: elasticsearch
  datahub_standalone_consumers_enabled: true

  sql:
    datasource:
      host: "${mysql_server}:3306"
      hostForMysqlClient: "${mysql_server}"
      port: "3306"
      url: "jdbc-secretsmanager:mysql://${mysql_server}:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2"
      driver: "com.amazonaws.secretsmanager.sql.AWSSecretsManagerMySQLDriver"
      username: "${environment}/datahub/mysql/datahub-db/datahub"
Please notice that 1. driver is changed to com.amazonaws.secretsmanager.sql.AWSSecretsManagerMySQLDriver 2. username is changed to secret ID in AWS secret manager 3. And Datasource URL now starts with "jdbc-secretsmanagermysql" Now this driver will pick username and password from AWS secret manager while making connection to database. This library acts like proxy between actual driver and client. And just insert username and password to connection properties from AWS secret manager. Why we need it? Our database user credentials (username/password) keep rotating after every few days. And they are stored in AWS secret-manager. Since credentials are rotating, we can not pass them to GMS container as env variables. Because GMS container stops working when DB user password is rotated. And GMS container does not crash. It just keeps trying to reconnect to database. And we see the following exception in logs
Copy code
10:07:34.031 [Thread-658] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher
java.util.concurrent.CompletionException: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<db>)(port=3306)(type=master) : Could not connect to <db>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.persistence.PersistenceException: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<db>)(port=3306)(type=master) : Could not connect to <db>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:35)
	at io.ebeaninternal.server.transaction.TransactionManager.createQueryTransaction(TransactionManager.java:360)
	at io.ebeaninternal.server.core.DefaultServer.createQueryTransaction(DefaultServer.java:2306)
	at io.ebeaninternal.server.core.OrmQueryRequest.initTransIfRequired(OrmQueryRequest.java:282)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1595)
	at io.ebeaninternal.server.core.DefaultServer.findList(DefaultServer.java:1574)
	at io.ebeaninternal.server.querydefn.DefaultOrmQuery.findList(DefaultOrmQuery.java:1481)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGetUnion(EbeanAspectDao.java:359)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:279)
	at com.linkedin.metadata.entity.ebean.EbeanAspectDao.batchGet(EbeanAspectDao.java:260)
	at com.linkedin.metadata.entity.EntityService.getEnvelopedAspects(EntityService.java:1504)
	at com.linkedin.metadata.entity.EntityService.getCorrespondingAspects(EntityService.java:353)
	at com.linkedin.metadata.entity.EntityService.getLatestEnvelopedAspects(EntityService.java:307)
	at com.linkedin.metadata.entity.EntityService.getEntitiesV2(EntityService.java:263)
	at com.linkedin.entity.client.JavaEntityClient.batchGetV2(JavaEntityClient.java:106)
	at com.linkedin.datahub.graphql.resolvers.MeResolver.lambda$get$0(MeResolver.java:55)
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
	... 1 common frames omitted
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=<db>)(port=3306)(type=master) : Could not connect to <db>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:192)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1346)
	at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:634)
	at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:150)
	at org.mariadb.jdbc.Driver.connect(Driver.java:89)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:208)
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:529)
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:524)
	at io.ebean.datasource.pool.ConnectionPool.createConnectionForQueue(ConnectionPool.java:766)
	at io.ebean.datasource.pool.PooledConnectionQueue._getPooledConnection(PooledConnectionQueue.java:314)
	at io.ebean.datasource.pool.PooledConnectionQueue.getPooledConnection(PooledConnectionQueue.java:270)
	at io.ebean.datasource.pool.ConnectionPool.getPooledConnection(ConnectionPool.java:817)
	at io.ebean.datasource.pool.ConnectionPool.getConnection(ConnectionPool.java:805)
	at io.ebeaninternal.server.transaction.TransactionFactoryBasic.createQueryTransaction(TransactionFactoryBasic.java:28)
	... 17 common frames omitted
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to <db>:3306 : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:73)
	at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:183)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:566)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1341)
	... 30 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:152)
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1392)
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1300)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:435)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.sslWrapper(AbstractConnectProtocol.java:649)
	at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:532)
	... 31 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)
	at sun.security.validator.Validator.validate(Validator.java:271)
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
	... 43 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451)
	... 49 common frames omitted
@mammoth-bear-12532
I don't see how this will ensure that you find the jar on the classpath in the server.
Ohh! you mean this jar will not be available in /datahub/datahub-gms/bin/war.war ? Then, I need to learn more about datahub
In that case it might be simplest to just drop in this jar as a runtime jar into your deployment?
I am actually trying that. I have added jar to EKS GMS container. But I am struggling with adding it to class-path. Jetty is not able to find the the jar.
Also, on adding it separately in class path: I was thinking that it would be good this library is distributed with war dependancy itself. This driver can be used in other places also 1. datahub-mysql-setup-job: Although it is not critical unless datahub-mysql-setup-job is running at the same time when database user credentials are rotated 2. Any long ingestion job that connects with these databases to pull database metadata into datahub
Just added metadata-service/war/build.gradle Can you please approve workflow run? "5 workflows awaiting approval" I am wondering if MR CI/CD creates any images those I can use to test in our env before actually merging this MR
2 Views
Open in Slack
PreviousNext
Powered by