Hi team, I am getting this error while running dat...
# troubleshoot
f
Hi team, I am getting this error while running datahub on version
0.10.2
.
Copy code
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
I am using k8s with helm to setup local instance. Default
linkedin/datahub-gms
image works fine but if I try to build the image locally and then use it for instance setup it throws this error. (We haven't changed anything in datahub-gms)
📖 1
✅ 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? 2️⃣ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue?
a
You might want to try restarting the GMS container to see if that could fix things!
f
Tried it, still getting same issue
a
Hi, it seems like this is a more widespread issue from the update and we’re looking at it internally!
f
HI @astonishing-answer-96712 Do we have any resolution for this?
s
I have the same issue since yesterday
a
Hi all! @brainy-tent-14503 might be able to help here
s
Did you try to modify initDelaySeconds on readiness & liveness for gms ? It solves my issue yesterday
b
Might be helpful to see the full logs from GMS startup. What might be going on is that when you build locally the git version may not match the version being emitted by the datahub-upgrade job. Both sides should be emitting a message with the version + revision and GMS waits for this message. This allows the system-update process to make any changes to the ES indices and other updates that might need to take effect before GMS is started. If you are not modifying the data model then that reindex is not needed and for your experiments (assuming you’re on a recent comment) can be bypassed with this env
BOOTSTRAP_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false
f
tried both the solutions, still facing same issue
Copy code
net.UnknownHostException: track.datahubproject.io
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:507)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:602)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:266)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:373)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:207)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:193)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
	at com.mixpanel.mixpanelapi.MixpanelAPI.sendData(MixpanelAPI.java:134)
	at com.mixpanel.mixpanelapi.MixpanelAPI.sendMessages(MixpanelAPI.java:172)
	at com.mixpanel.mixpanelapi.MixpanelAPI.deliver(MixpanelAPI.java:107)
	at com.mixpanel.mixpanelapi.MixpanelAPI.deliver(MixpanelAPI.java:83)
	at com.mixpanel.mixpanelapi.MixpanelAPI.sendMessage(MixpanelAPI.java:71)
	at com.linkedin.gms.factory.telemetry.DailyReport.<init>(DailyReport.java:52)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory.dailyReport(ScheduledAnalyticsFactory.java:26)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$511f49aa.CGLIB$dailyReport$0(<generated>)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$511f49aa$$FastClassBySpringCGLIB$$9a66b25b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$511f49aa.dailyReport(<generated>)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.runner.Runner.run(Runner.java:519)
	at org.eclipse.jetty.runner.Runner.main(Runner.java:564)
can this cause this issue?
Copy code
Caused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
	at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.onError(ChannelPoolLifecycle.java:142)
	at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.lambda$create$0(ChannelPoolLifecycle.java:97)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Another error that I got
I also tried re-running system update job but still getting same error
a
Hmm, I think I missed the part where you might be running in an isolated environment. The DNS error is likely because you’re not resolving public domains. To resolve that error you’d need to probably turn off server telemetry using this env var. Using helm this would be an extraEnv on the gms subchart passed in here.
f
Still getting same error
b
You definitely disabled all those telemetry settings using environment variables? Is the stack trace different? I can see that lines referenced in the stack should not be present if disabled per this conditional, please double check the environment variable settings match expectations. If the error is now different please include that log message. Thanks!
g
@brainy-tent-14503 @astonishing-answer-96712 Each time when we run the upgrade from helm
DATAHUB_REVISION
increments and the same will update as the current system updated version in GMS. This won’t match the expected version of the datahub system version, it leads to gms being stuck with upgrading
Copy code
2023-07-11 09:41:33,225 [ThreadPoolTaskExecutor-1] INFO  c.l.m.b.k.DataHubUpgradeKafkaListener:92 - Latest system update version: v0.10.4-7
2023-07-11 09:41:33,225 [ThreadPoolTaskExecutor-1] WARN  c.l.m.b.k.DataHubUpgradeKafkaListener:99 - System version is not up to date: v0.10.4-0. Waiting for datahub-upgrade to complete...
https://github.com/datahub-project/datahub/blob/ea3e119210802cadc886dd28bafa7eda09[…]m/linkedin/metadata/boot/kafka/DataHubUpgradeKafkaListener.java It looks like it is a bug to me Deploying gms with
BOOTSTRAP_SYSTEM_UPDATE_WAIT_FOR_SYSTEM_UPDATE=false
solved the issue for us
a
Let me see if I understand what happens. The expectation is that we are upgrading datahub not from version A -> B, but A -> A and changing something in the values. In this case, the system update job will emit another message for that revision. However the values changed do not effect GMS and therefore the revision on this component is not incremented by helm. Second factor is that by design the GMS component is only looking for the latest message instead of looking from the beginning of the topic where supposedly there is an earlier message from the first revision 0. A second solution would involve a change to the helm chart to restart GMS and other components whenever an upgrade is performed even if the values for that deployment/pod is not changed. Another option is potentially allow the helm chart values to manually specify a revision. This manual revision would need to be incremented whenever reindexing might be needed (this includes various values around search index settings and any changes around custom entities). This needs some additional thought. Not waiting can potentially break GMS and other components whenever the indices need the reindex prior to starting and may lead to inconsistent documents in the indices.
g
I think a good solution here is whenever all upgrade task completes including restore-backup/restore-indices setup the expected version in gms
systemEnvironment['DATAHUB_REVISION']
to system update job revision and restart GMS and other components