https://datahubproject.io logo
Join Slack
Powered by
<@U03MF8MU5P0> Apologies to get back on yesterday'...
# troubleshoot
n
@brainy-tent-14503 Apologies to get back on yesterday's issue again. Last evening I was able to login to Datahub 
/login
using admin creds.. But, now its throwing 
Caused by: java.lang.RuntimeException: Failed to generate session token for user
error Error stack in 🧵
Copy code
! @8038j1785 - Internal server error, for (POST) [/logIn] ->

play.api.UnexpectedException: Unexpected exception[RuntimeException: Failed to generate session token for user]
	at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:340)
	at play.api.http.DefaultHttpErrorHandler.onServerError(HttpErrorHandler.scala:263)
	at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:443)
	at play.core.server.AkkaHttpServer$$anonfun$1.applyOrElse(AkkaHttpServer.scala:441)
	at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.RuntimeException: Failed to generate session token for user
	at client.AuthServiceClient.generateSessionTokenForUser(AuthServiceClient.java:101)
	at controllers.AuthenticationController.logIn(AuthenticationController.java:154)
	at router.Routes$$anonfun$routes$1.$anonfun$applyOrElse$14(Routes.scala:532)
	at play.core.routing.HandlerInvokerFactory$$anon$8.resultCall(HandlerInvoker.scala:147)
	at play.core.routing.HandlerInvokerFactory$$anon$8.resultCall(HandlerInvoker.scala:146)
	at play.core.routing.HandlerInvokerFactory$JavaActionInvokerFactory$$anon$3$$anon$4$$anon$5.invocation(HandlerInvoker.scala:112)
	at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:128)
	at play.mvc.Action.lambda$call$0(Action.java:89)
	at java.base/java.util.Optional.map(Optional.java:265)
	at play.mvc.Action.call(Action.java:81)
	at play.http.DefaultActionCreator$1.call(DefaultActionCreator.java:33)
	at play.core.j.JavaAction.$anonfun$apply$8(JavaAction.scala:188)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)
	at scala.util.Success.map(Try.scala:213)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:77)
	at play.api.libs.streams.Execution$trampoline$.execute(Execution.scala:70)
	at play.core.j.HttpExecutionContext.execute(HttpExecutionContext.scala:69)
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
	at scala.concurrent.impl.Promise$KeptPromise$Kept.onComplete(Promise.scala:372)
	at scala.concurrent.impl.Promise$KeptPromise$Kept.onComplete$(Promise.scala:371)
	at scala.concurrent.impl.Promise$KeptPromise$Successful.onComplete(Promise.scala:379)
	at scala.concurrent.impl.Promise.transform(Promise.scala:33)
	at scala.concurrent.impl.Promise.transform$(Promise.scala:31)
	at scala.concurrent.impl.Promise$KeptPromise$Successful.transform(Promise.scala:379)
	at scala.concurrent.Future.map(Future.scala:292)
	at scala.concurrent.Future.map$(Future.scala:292)
	at scala.concurrent.impl.Promise$KeptPromise$Successful.map(Promise.scala:379)
	at scala.concurrent.Future$.apply(Future.scala:659)
	at play.core.j.JavaAction.apply(JavaAction.scala:189)
	at play.api.mvc.Action.$anonfun$apply$2(Action.scala:95)
	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
	... 13 common frames omitted
Caused by: java.lang.RuntimeException: Bad response from the Metadata Service: HTTP/1.1 503 Service Unavailable ResponseEntityProxy{[Content-Type: text/html;charset=iso-8859-1,Content-Length: 516,Chunked: false]}
	at client.AuthServiceClient.generateSessionTokenForUser(AuthServiceClient.java:97)
	... 48 common frames omitted
I am not sure if any AUTH_OIDC env vars should be changed or added....
Copy code
AUTH_OIDC_ENABLED=true
AUTH_OIDC_DISCOVERY_URI=https://<org>.<http://okta.com/.well-known/openid-configuration|okta.com/.well-known/openid-configuration>
AUTH_OIDC_SCOPE=openid profile email groups
AUTH_OIDC_EXTRACT_GROUPS_ENABLED=true
AUTH_OIDC_USER_NAME_CLAIM=email
AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+)
AUTH_NATIVE_ENABLED=false
AUTH_OIDC_BASE_URL=https://<url>.<http://dev.k8s.pelotime.com|dev.k8s.pelotime.com>
The metadata service does respond when hit on /health endpoint.
a
Hi @nutritious-bird-77396, did you recently update to 0.9.4? If so, this is a bug we are currently working on patching
o
Can you check your gms frotend debug log to see if you are also getting a pac4j error related to pkce/code verifier being invalid?
n
I am working on updating to 
0.9.3
version. No GMS logs don't have any errors...
Is this issue existing in 
0.9.3
as well? Then should i wait and install the new patch release when its out..
d
No GMS logs don’t have any errors…
the errors would be in your datahub-frontend logs
b
The error here is the connection between the frontend and gms. Please use the same tag on GMS as used for the frontend and then we should see some logs from GMS.
n
Got it....Updating GMS image as well to
Copy code
images:
  - name: linkedin/datahub-gms
    newTag: 2de9d3d
@brainy-tent-14503 Throws RemoteInvocationError now...
Copy code
2022-12-21 20:15:19,353 [application-akka.actor.default-dispatcher-33] ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature
2022-12-21 20:15:29,635 [application-akka.actor.default-dispatcher-33] ERROR auth.sso.oidc.OidcCallbackLogic - Failed to perform post authentication steps. Redirecting to error page.
com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-gms-service.data-platform.svc.cluster.local:8080/aspects>
	at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)
	at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponse(ResponseFutureImpl.java:94)
	at com.linkedin.common.client.BaseClient.sendClientRequest(BaseClient.java:36)
	at com.linkedin.entity.client.RestliEntityClient.ingestProposal(RestliEntityClient.java:632)
	at com.linkedin.entity.client.EntityClient.ingestProposal(EntityClient.java:282)
	at auth.sso.oidc.OidcCallbackLogic.setUserStatus(OidcCallbackLogic.java:429)
	at auth.sso.oidc.OidcCallbackLogic.handleOidcCallback(OidcCallbackLogic.java:141)
	at auth.sso.oidc.OidcCallbackLogic.perform(OidcCallbackLogic.java:104)
	at controllers.SsoCallbackController$SsoCallbackLogic.perform(SsoCallbackController.java:83)
	at controllers.SsoCallbackController$SsoCallbackLogic.perform(SsoCallbackController.java:69)
	at org.pac4j.play.CallbackController.lambda$callback$0(CallbackController.java:56)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at play.core.j.HttpExecutionContext$$anon$2.run(HttpExecutionContext.scala:77)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://datahub-gms-service.data-platform.svc.cluster.local:8080/aspects>
	at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:67)
	at com.linkedin.r2.transport.http.client.rest.ExecutionCallback.lambda$onResponse$0(ExecutionCallback.java:64)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.TimeoutException: Exceeded request timeout of 10000ms
	at com.linkedin.r2.transport.http.client.TimeoutTransportCallback$1.run(TimeoutTransportCallback.java:69)
	at com.linkedin.r2.util.Timeout.lambda$new$0(Timeout.java:77)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	... 3 common frames omitted
o
That's a timeout, is gms running at that location? 
kubectl exec -it <pod-name> -- nslookup  datahub-gms-service.data-platform.svc.cluster.local
Were you able to confirm the service was running here Arun?
n
Thanks for checking in Ryan. Yes, the gms service is running I am able to hit with the external domain successfully but the internal (namespace) url doesn't work when using the above nslookup command. So reached out to the internal compute team on it. Will get back on this once i get a response. have a nice holidays!
49 Views
Open in Slack
PreviousNext
Powered by