https://datahubproject.io logo
Join Slack
Powered by
Hi guys, I have trouble getting Datahub running on...
# troubleshoot
f
Hi guys, I have trouble getting Datahub running on K8s with existing backends. I think I finally got existing Postgres and Kafka DBs working with Datahub, but I'm still getting weird errors with Elasticsearch. elasticsearch-setup-job logs show status 200
Copy code
2022/02/25 10:06:16 Waiting for: <http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>
2022/02/25 10:06:16 Problem with request: Get <http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>: dial tcp 10.152.183.143:9200: connect: connection refused. Sleeping 1s
2022/02/25 10:06:17 Received 200 from <http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>

datahub_usage_event_policy exists

datahub_usage_event_index_template exists
2022/02/25 10:06:17 Command finished successfully.
But GMS logs say
Copy code
09:36:17.790 [main] WARN  o.s.w.c.s.XmlWebApplicationContext:558 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'metadataChangeLogProcessor' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-1251704996605983937.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/MetadataChangeLogProcessor.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'updateIndicesHook' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-1251704996605983937.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/hook/UpdateIndicesHook.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is ElasticsearchStatusException[method [HEAD], host [<http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]]; nested: ResponseException[method [HEAD], host [<http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]];
09:36:17.792 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested
09:36:17.792 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down
09:36:17.795 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools
09:36:17.796 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown
09:36:17.797 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient:249 - Shutdown requested
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.AbstractNettyClient:252 - Shutting down
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools
09:36:17.805 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown
09:36:17.806 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels
09:36:17.806 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:152 - Shutting down 0 connection pools
09:36:17.806 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:162 - All connection pools shutdown
09:36:17.807 [main] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:119 - All connection pools shut down, closing all channels
Feb 25, 2022 9:36:17 AM org.neo4j.driver.internal.logging.JULogger info
INFO: Closing driver instance 868394850
09:36:17.807 [R2 Nio Event Loop-1-1] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete
09:36:17.807 [R2 Nio Event Loop-1-2] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete
09:36:17.809 [R2 Nio Event Loop-1-1] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete
09:36:17.810 [R2 Nio Event Loop-1-2] INFO  c.l.r.t.h.c.c.ChannelPoolManagerImpl:103 - Shutdown complete
09:36:18.027 [main] INFO  o.a.k.clients.producer.KafkaProducer:1182 - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
09:36:18.027 [main] INFO  o.a.k.clients.producer.KafkaProducer:1182 - [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
09:36:18.051 [main] ERROR o.s.web.context.ContextLoader:313 - Context initialization failed
Copy code
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'metadataChangeLogProcessor' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-1251704996605983937.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/MetadataChangeLogProcessor.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'updateIndicesHook' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-1251704996605983937.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/hook/UpdateIndicesHook.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is ElasticsearchStatusException[method [HEAD], host [<http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]]; nested: ResponseException[method [HEAD], host [<http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]];
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:798)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:228)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:921)
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:554)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:888)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:357)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1443)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1407)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:821)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:276)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
        at org.eclipse.jetty.server.Server.start(Server.java:407)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:106)
        at org.eclipse.jetty.server.Server.doStart(Server.java:371)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
        at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
        at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
Copy code
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'updateIndicesHook' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-1251704996605983937.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/hook/UpdateIndicesHook.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is ElasticsearchStatusException[method [HEAD], host [<http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]]; nested: ResponseException[method [HEAD], host [<http://elastic-cluster-es-http.elastic-system.svc.cluster.local:9200>], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]];
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:314)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:295)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1358)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1204)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1287)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1207)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:885)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:789)
        ... 41 common frames omitted
l
@early-lamp-41924 ^
e
It’s throwing “401 Unauthorized” can you make sure the credentials are correct and that the account you are using to access the elasticsearch cluster has permissions to create indices in the cluster?
10 Views
Open in Slack
PreviousNext
Powered by