fierce-electrician-85924
04/14/2023, 10:30 AM0.10.2 .
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
I am using k8s with helm to setup local instance. Default linkedin/datahub-gms image works fine but if I try to build the image locally and then use it for instance setup it throws this error. (We haven't changed anything in datahub-gms)lively-cat-88289
04/14/2023, 10:30 AMastonishing-answer-96712
04/14/2023, 5:48 PMfierce-electrician-85924
04/17/2023, 1:17 PMastonishing-answer-96712
04/18/2023, 4:47 PMfierce-electrician-85924
05/16/2023, 5:57 AMsalmon-garden-47148
05/16/2023, 6:08 AMaloof-gpu-11378
05/16/2023, 9:26 PMsalmon-garden-47148
05/17/2023, 6:53 AMbrainy-tent-14503
05/17/2023, 5:13 PMBOOTSTRAP_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=falsefierce-electrician-85924
05/18/2023, 12:57 PMfierce-electrician-85924
05/18/2023, 12:58 PMnet.UnknownHostException: track.datahubproject.io
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
at com.mixpanel.mixpanelapi.MixpanelAPI.sendData(MixpanelAPI.java:134)
at com.mixpanel.mixpanelapi.MixpanelAPI.sendMessages(MixpanelAPI.java:172)
at com.mixpanel.mixpanelapi.MixpanelAPI.deliver(MixpanelAPI.java:107)
at com.mixpanel.mixpanelapi.MixpanelAPI.deliver(MixpanelAPI.java:83)
at com.mixpanel.mixpanelapi.MixpanelAPI.sendMessage(MixpanelAPI.java:71)
at com.linkedin.gms.factory.telemetry.DailyReport.<init>(DailyReport.java:52)
at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory.dailyReport(ScheduledAnalyticsFactory.java:26)
at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$511f49aa.CGLIB$dailyReport$0(<generated>)
at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$511f49aa$$FastClassBySpringCGLIB$$9a66b25b.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$511f49aa.dailyReport(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.runner.Runner.run(Runner.java:519)
at org.eclipse.jetty.runner.Runner.main(Runner.java:564)
can this cause this issue?fierce-electrician-85924
05/18/2023, 12:58 PMCaused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.onError(ChannelPoolLifecycle.java:142)
at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.lambda$create$0(ChannelPoolLifecycle.java:97)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Another error that I gotfierce-electrician-85924
05/18/2023, 1:13 PMaloof-gpu-11378
05/18/2023, 8:24 PMfierce-electrician-85924
05/23/2023, 12:09 PMbrainy-tent-14503
05/23/2023, 7:55 PMgray-ocean-32209
07/12/2023, 5:11 PMDATAHUB_REVISION increments and the same will update as the current system updated version in GMS. This won’t match the expected version of the datahub system version, it leads to gms being stuck with upgrading
2023-07-11 09:41:33,225 [ThreadPoolTaskExecutor-1] INFO c.l.m.b.k.DataHubUpgradeKafkaListener:92 - Latest system update version: v0.10.4-7
2023-07-11 09:41:33,225 [ThreadPoolTaskExecutor-1] WARN c.l.m.b.k.DataHubUpgradeKafkaListener:99 - System version is not up to date: v0.10.4-0. Waiting for datahub-upgrade to complete...
https://github.com/datahub-project/datahub/blob/ea3e119210802cadc886dd28bafa7eda09[…]m/linkedin/metadata/boot/kafka/DataHubUpgradeKafkaListener.java
It looks like it is a bug to me
Deploying gms with BOOTSTRAP_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false solved the issue for usaloof-gpu-11378
07/12/2023, 6:30 PMgray-ocean-32209
07/13/2023, 2:31 PMsystemEnvironment['DATAHUB_REVISION']
to system update job revision and restart GMS and other components