https://datahubproject.io logo
Join Slack
Powered by
Hi all, working on a POC of Datahub. Running into ...
# getting-started
f
Hi all, working on a POC of Datahub. Running into some issues with connecting to our external Kafka provider, as it uses SSL and Basic Auth for schema registry. Was hoping there would a way to configure consumers to use SSL via environment vairables, but it looks like the Spring Kafka library doesn't support these specific configs. There's some stackoverflow stuff that implies they at least don't have it for SSL https://stackoverflow.com/questions/51316017/spring-boot-spring-kafka-ssl-configuration-by-environment-variables-impossible So it would seem that this configuration has to be done here https://github.com/linkedin/datahub/blob/master/metadata-jobs/mae-consumer-job/src/main/resources/application.properties Has anybody else run into this? I'm thinking of testing out a fix locally and opening a PR, would love to know if I'm missing soemthing
p
f
Oh interesting. I had seen the Spring kafka pass-passthrough environment variables, but couldn't find any docs on which ones were supported. This is promising though, thanks!
b
Which company are you with Sam, if you don't mind sharing?
f
SpotHero!
b
Oh of course no wonder the name was familiar. Let us know if you come across any other issues.
m
Sam please let us know if this still doesn't support SSL configurations and we can look into it
Also, let us know if the inverse is true (it does support it)
Would be good to know 🙂
f
The SSL configuration is working now, with essentially the approach documented here. I guess I don’t understand Spring Boot super well, was expecting to be able to find an explicit set of supported configs somewhere in their docs, but it appears that instead it just passes in anything it finds with the prefix KAFKA_PROPERTIES
m
Interesting, thanks
We should add this to our documentation somewhere 😛
did you find docs on this we can link to?
f
No I was really just guessing based off the link @plain-arm-6774 posted above. I dug through here a bit but didn’t find anything related. https://docs.spring.io/spring-kafka/docs/2.5.5.RELEASE/reference/html/
It seems that Spring Boot just takes all environment variables that are prefixed with Spring and makes them available as global properties?
Not to pile on to this thread, but it seems that I now have the two consumers and the frontend running properly, but the GMS service is crashing on startup with a null pointer exception.
Copy code
19:30:08.912 [main] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'datasetDao' defined in com.linkedin.dataset.factory.DatasetDaoFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.dao.EbeanLocalDAO]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException
m
can you start a new thread and paste the full stack trace there
also, here's documentation on the environment variable to system property thing spring is doing
f
Copy code
19:30:08.912 [main] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'datasetDao' defined in com.linkedin.dataset.factory.DatasetDaoFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.dao.EbeanLocalDAO]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:484) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401) ~[spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103) [spring-web-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:921) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:554) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:888) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:357) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1443) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1407) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.Server.start(Server.java:407) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.server.Server.doStart(Server.java:371) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.runner.Runner.run(Runner.java:520) [jetty-runner.jar:9.4.20.v20190813]
	at org.eclipse.jetty.runner.Runner.main(Runner.java:565) [jetty-runner.jar:9.4.20.v20190813]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.dao.EbeanLocalDAO]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	... 41 more
Caused by: java.lang.NullPointerException
	at io.ebean.datasource.pool.ConnectionPool.notifyDataSourceIsDown(ConnectionPool.java:406) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:535) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:524) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.pool.ConnectionPool.createConnectionForQueue(ConnectionPool.java:766) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.pool.PooledConnectionQueue.ensureMinimumConnections(PooledConnectionQueue.java:227) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.pool.ConnectionPool.initialise(ConnectionPool.java:301) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.pool.ConnectionPool.<init>(ConnectionPool.java:246) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebean.datasource.core.Factory.createPool(Factory.java:15) ~[ebean-datasource-4.3.3.jar:?]
	at io.ebeaninternal.server.core.DefaultContainer.getDataSourceFromConfig(DefaultContainer.java:273) ~[ebean-11.33.3.jar:?]
	at io.ebeaninternal.server.core.DefaultContainer.setDataSource(DefaultContainer.java:217) ~[ebean-11.33.3.jar:?]
	at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:103) ~[ebean-11.33.3.jar:?]
	at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:35) ~[ebean-11.33.3.jar:?]
	at io.ebean.EbeanServerFactory.createInternal(EbeanServerFactory.java:109) ~[ebean-11.33.3.jar:?]
	at io.ebean.EbeanServerFactory.create(EbeanServerFactory.java:70) ~[ebean-11.33.3.jar:?]
	at com.linkedin.metadata.dao.EbeanLocalDAO.createServer(EbeanLocalDAO.java:103) ~[ebean-dao.jar:?]
	at com.linkedin.metadata.dao.EbeanLocalDAO.<init>(EbeanLocalDAO.java:78) ~[ebean-dao.jar:?]
	at com.linkedin.dataset.factory.DatasetDaoFactory.createInstance(DatasetDaoFactory.java:32) ~[factories.jar:?]
	at com.linkedin.dataset.factory.DatasetDaoFactory$$EnhancerBySpringCGLIB$$d0836bd2.CGLIB$createInstance$0(<generated>) ~[factories.jar:?]
	at com.linkedin.dataset.factory.DatasetDaoFactory$$EnhancerBySpringCGLIB$$d0836bd2$$FastClassBySpringCGLIB$$88d8dc47.invoke(<generated>) ~[factories.jar:?]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at com.linkedin.dataset.factory.DatasetDaoFactory$$EnhancerBySpringCGLIB$$d0836bd2.createInstance(<generated>) ~[factories.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
	... 41 more
is this closer to what you’re looking for? apologies for the wall of text
m
is mysql running?
f
Mysql is definitely running, although there’s a chance the service is having trouble connecting
just above the stack trace I see this
Copy code
19:30:07.777 [main] INFO  o.a.k.clients.producer.KafkaProducer - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
m
how are you running datahub? quickstart.sh? Let me see if I can reproduce
f
I’m running a modified version of the helm chart in contrib, running managed ES and MySQL in AWS, and a neo4j install in k8s
My current suspicion is that the kafka producer is failing to connect to schema registry, because I know that needs basic auth but haven’t been able to pass those configs in
By I haven’t seen any explicit logs about that
m
Considering the stack trace is in the ebean setup, I'd assume it is a mysql issue, but don't know for sure
NPE doesn't give much information, sadly
f
ok thats actually good direction, we’re trying to validate that we can used managed db’s for this, so the networking has a lot of moving pieces atm
let me take a better look at how the mysql is configured
I’m not finding anything obvious wrong with the db unforunately
No other piece of the infrastructure talks to MySQL?
m
no afaik. Should just be the GMS' ebean DAO
worth noting the exception is in 
notifyDataSourceIsDown
lol
so seems like a my sql issue
f
Yep it was a db issue, got it sorted this morning. TLDR: bad credential
Thanks for all the help!
m
No problem! Glad to hear it is fixed.
s
@fast-exabyte-18411 I am currently getting the same issue, mind sharing how you fixed this problem? For me mysql is running correctly.
Open in Slack
PreviousNext
Powered by