https://datahubproject.io logo
Join Slack
Powered by
<@U03MF8MU5P0> As discussed this morning in the of...
# troubleshoot
n
@brainy-tent-14503 As discussed this morning in the office hours.. I am using 
v0.9.3
version of datahub-gms and 
v0.9.3
datahub-frontend and 
0.0.8
version of datahub actions. I see the below message when i hit the datahub frontend endpoint - 
Failed to perform post authentication steps. Error message: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-gms-service>.<team>.svc.cluster.local:8080/aspects
When hitting gms health endpoint returns 200. Also, datahub frontend /admin endpoint returns 
GOOD
Copy code
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
11
SLF4J: Defaulting to no-operation (NOP) logger implementation
10
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
9
WARNING: An illegal reflective access operation has occurred
8
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/datahub-frontend/lib/com.google.inject-guice-4.2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
7
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
6
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
5
WARNING: All illegal access operations will be denied in a future release
4
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
3
SLF4J: Defaulting to no-operation MDCAdapter implementation.
2
SLF4J: See <http://www.slf4j.org/codes.html#no_static_mdc_binder> for further details.
1
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
Log message from the pod. It doesn't show the reason for the error
b
I believe this is actually not hung what does 
/index.html
return?
n
It takes to the same error as well
Just to let you know we had 
AUTH_OIDC_ENABLED
= true
b
This is not typical for the gms hostname, usually its not fully qualified like this. 
<http://datahub-gms-service>.<team>.svc.cluster.local:8080
If its in the same namespace and assuming you’ve configured the service as datahub-gms-service and not the typical name
In short, is there connectivity between the frontend pod and that hostname and port?
I assume you added the 
<team>
manually otherwise that’s not a correct dns name
n
Yes i added team manually to not expose the complete url
b
ok, that’s what I assumed
n
Its just an internal cluster url
b
Right, its typical when running in different namespaces
n
When i hit it separately using health endpoint it returns 200
Both services are running in same namespace
How can i confirm if there is connectivity between the frontend pod and that hostname and port?
let me login to the pod..one sec
b
Let me get you a command to run.
Assuming you have exec access, run the health check from the frontend using the exact hostname
Copy code
curl -v <http://datahub-gms-service>.<team>.svc.cluster.local:8080/health
This would check dns resolution and port connectivity from the frontend. I’ve certainly seen cases where dns isn’t working as expected from within the pod
n
Copy code
curl -v <http://datahub-gms-service.data-platform.svc.cluster.local:8080/health>
*   Trying <IP-Address>:8080...
* Connected to datahub-gms-service.data-platform.svc.cluster.local (<IP-Address>) port 8080 (#0)
> GET /health HTTP/1.1
> Host: datahub-gms-service.data-platform.svc.cluster.local:8080
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Tue, 20 Dec 2022 22:12:50 GMT
< Content-Length: 0
< Server: Jetty(9.4.46.v20220331)
<
* Connection #0 to host datahub-gms-service.data-platform.svc.cluster.local left intact
it looks like the connection seems working right?
b
Yes, working
Let me see if I can find that error string somewhere.
Definitely from the oidc code
Can you share your oidc configuration environment variables? redacting anything sensitive is fine, like the client id/secret
This envs start with 
AUTH_OIDC
n
Copy code
AUTH_OIDC_ENABLED=true
AUTH_OIDC_DISCOVERY_URI=https://<okta-domain>/.well-known/openid-configuration
AUTH_OIDC_SCOPE=openid profile email groups
AUTH_OIDC_EXTRACT_GROUPS_ENABLED=true
AUTH_OIDC_USER_NAME_CLAIM=email
AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+)
AUTH_NATIVE_ENABLED=false
AUTH_OIDC_BASE_URL=<https://datahub>.<domain-url>
As you know nothing changed on the OIDC front as it worked in the old version..
Its the Readiness probe and liveness probe that fails the app
Even /admin endpoint returns 
GOOD
b
There are some changes perhaps in the OIDC logic, I am looking through the code for oidc, try to determine where it might be throwing that exception. There is some branching logic. What do you mean the readiness/liveness proves fails the app? It should be exercising /admin (idk why its named that). Are you seeing k8 restarting the pod?
I am wondering about these oidc settings specifically as they control the branching in that section of code.
I am assuming all defaults
However I do see groups added to scope but extract groups is false by default
n
Yes, all except 
AUTH_OIDC_EXTRACT_GROUPS_ENABLED
are default Was this a recent addition 
AUTH_OIDC_GROUPS_CLAIM
?
b
It defaults to 
groups
so it is also fine
Ok, lets see if I can pull out a different image to try in the frontend, I think we’re missing some logging messages for sure.
n
Yes, k8s is restarting the pod because its unhealthy
b
Can you confirm that the probe is still set to /admin ?
n
Yes, it is!
b
that hasn’t changed in a very long time …
Ok, so I’d like to try swapping the frontend image with 
linkedin/datahub-frontend-react:2de9d3d
Specifically with trying to get more logs, that image was built from this commit.
n
is it still compatible with 
v0.9.3
gms version?
b
yes, it was committed just after that release
There is also a release being cut today, if you want to wait for v0.9.4
n
Sure, I am only deploying to dev definitely will upgrade. But, i am not sure if the issue would persist there too
Now, deploying with the image you provided - 
linkedin/datahub-frontend-react:2de9d3d
b
I would expect to see more logging at least. Also I’d be interested in restarts after this deploy.
n
Yup this gives better logging... As suspected its to do with Kafka Connection..
Copy code
1) Error injecting constructor, org.apache.kafka.common.KafkaException: Failed to construct kafka producer
77
  at controllers.TrackingController.<init>(TrackingController.java:50)
76
  while locating controllers.TrackingController
75
    for the 7th parameter of router.Routes.<init>(Routes.scala:45)
74
  while locating router.Routes
73
  while locating play.api.inject.RoutesProvider
72
  while locating play.api.routing.Router
71
70
1 error
69
	at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:226)
68
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1053)
67
	at play.api.http.DefaultHttpRequestHandler.routeRequest(HttpRequestHandler.scala:302)
66
	at play.api.http.JavaCompatibleHttpRequestHandler.routeRequest(HttpRequestHandler.scala:417)
65
	at play.api.http.DefaultHttpRequestHandler.routeWithFallback$1(HttpRequestHandler.scala:200)
64
	at play.api.http.DefaultHttpRequestHandler.handlerForRequest(HttpRequestHandler.scala:250)
63
	at play.core.server.Server$.getHandlerFor(Server.scala:117)
62
	... 22 common frames omitted
61
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
60
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:433)
59
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
58
	at controllers.TrackingController.createKafkaProducer(TrackingController.java:138)
57
	at controllers.TrackingController.<init>(TrackingController.java:55)
56
	at controllers.TrackingController$$FastClassByGuice$$c3b1bcca.newInstance(<generated>)
55
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)
54
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
53
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
52
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
51
	at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
50
	at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
49
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
48
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
47
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
46
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
45
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1086)
44
	at play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:435)
43
	at play.api.inject.ContextClassLoaderInjector.$anonfun$instanceOf$3(Injector.scala:116)
42
	at play.api.inject.ContextClassLoaderInjector.withContext(Injector.scala:124)
41
	at play.api.inject.ContextClassLoaderInjector.instanceOf(Injector.scala:116)
40
	at play.api.inject.RoutesProvider.$anonfun$get$2(BuiltinModule.scala:116)
39
	at scala.Option.fold(Option.scala:251)
38
	at play.api.inject.RoutesProvider.get$lzycompute(BuiltinModule.scala:116)
37
	at play.api.inject.RoutesProvider.get(BuiltinModule.scala:111)
36
	at play.api.inject.RoutesProvider.get(BuiltinModule.scala:105)
35
	at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
34
	at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77)
33
	at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59)
32
	at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
31
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050)
30
	... 27 common frames omitted
29
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts of type JKS
28
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160)
27
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146)
26
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
25
	at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99)
24
	at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:441)
23
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:422)
22
	... 56 common frames omitted
21
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts of type JKS
20
	at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:160)
19
	at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:102)
18
	at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:93)
17
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:156)
16
	... 61 common frames omitted
15
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts of type JKS
14
	at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
13
	at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:153)
12
	... 64 common frames omitted
11
Caused by: java.nio.file.NoSuchFileException: /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts
10
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
9
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
8
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
7
	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
6
	at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
5
	at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
4
	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
3
	at java.base/java.nio.file.Files.newInputStream(Files.java:156)
2
	at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:282)
1
	... 65 common frames omitted
b
Interesting, I thought we moved to jdk11
n
I think it could be because of this config i have..
Copy code
KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts
Need to find what would be the location in jdk11
b
/usr/lib/jvm/java-11-openjdk/lib/security/cacerts
That’s actually a symlink to 
/etc/ssl/certs/java/cacerts
n
After setting the new configs... don't see any error in application logs..
Copy code
ssl.truststore.location = /usr/lib/jvm/java-11-openjdk/lib/security/cacerts
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	<http://transaction.timeout.ms|transaction.timeout.ms> = 60000
	transactional.id = null
	value.serializer = class org.apache.kafka.common.serialization.StringSerializer

2022-12-20 23:12:37,612 [application-akka.actor.default-dispatcher-10] INFO  o.a.k.c.s.a.AbstractLogin - Successfully logged in.
2022-12-20 23:12:37,635 [application-akka.actor.default-dispatcher-10] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 2.3.0
2022-12-20 23:12:37,636 [application-akka.actor.default-dispatcher-10] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: fc1aaa116b661c8a
2022-12-20 23:12:37,636 [application-akka.actor.default-dispatcher-10] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1671577957634
2022-12-20 23:12:37,959 [kafka-producer-network-thread | datahub-frontend] INFO  org.apache.kafka.clients.Metadata - [Producer clientId=datahub-frontend] Cluster ID: XMaEhWiXSv-slXr_qUzWoQ
But, the frontend still complains the same error....Tried it in an incognito window still the same result
Looks like an OIDC Callback issue..
Copy code
2022-12-20 23:21:33,269 [application-akka.actor.default-dispatcher-98] ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature
2022-12-20 23:21:43,574 [application-akka.actor.default-dispatcher-98] ERROR auth.sso.oidc.OidcCallbackLogic - Failed to perform post authentication steps. Redirecting to error page.
com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-gms-service.data-platform.svc.cluster.local:8080/aspects>
Its working now! Not sure why it took a while. Thanks a lot @brainy-tent-14503
15 Views
Open in Slack
PreviousNext
Powered by