Hi ! Setting up Datahub on EKS with Azure based SSO. Getting an error about frontend not authorized with gms. client id & secret doesn't match / missing 'Bearer' prefix ... anyone can shed some light ?

datahub-gms-6698965898-bjv4x datahub-gms 2023-05-04 14:10:38,024 [qtp447981768-278] WARN  c.d.a.a.AuthenticatorChain:80 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Provided credentials do not match known system client id & client secret. Check your configuration values...), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Authorization header missing 'Bearer' prefix.)]
datahub-gms-6698965898-bjv4x datahub-gms 2023-05-04 14:10:40,028 [qtp447981768-269] WARN  c.d.a.a.AuthenticatorChain:80 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Provided credentials do not match known system client id & client secret. Check your configuration values...), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Authorization header missing 'Bearer' prefix.)]
datahub-frontend-7b758459b7-vs8sj datahub-frontend 2023-05-04 14:10:44,033 [application-akka.actor.default-dispatcher-12] ERROR auth.sso.oidc.OidcCallbackLogic - Failed to perform post authentication steps. Redirecting to error page.
datahub-frontend-7b758459b7-vs8sj datahub-frontend java.lang.RuntimeException: Failed to provision user with urn urn:li:corpuser:me@company.com.
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at auth.sso.oidc.OidcCallbackLogic.tryProvisionUser(OidcCallbackLogic.java:340)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at auth.sso.oidc.OidcCallbackLogic.handleOidcCallback(OidcCallbackLogic.java:129)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at auth.sso.oidc.OidcCallbackLogic.perform(OidcCallbackLogic.java:107)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at controllers.SsoCallbackController$SsoCallbackLogic.perform(SsoCallbackController.java:89)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at controllers.SsoCallbackController$SsoCallbackLogic.perform(SsoCallbackController.java:75)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at org.pac4j.play.CallbackController.lambda$callback$0(CallbackController.java:54)
datahub-gms-6698965898-bjv4x datahub-gms 2023-05-04 14:10:44,031 [qtp447981768-277] WARN  c.d.a.a.AuthenticatorChain:80 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Provided credentials do not match known system client id & client secret. Check your configuration values...), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Authorization header missing 'Bearer' prefix.)]
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at 
...
auth.sso.oidc.OidcCallbackLogic.tryProvisionUser(OidcCallbackLogic.java:321)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	... 14 common frames omitted
datahub-frontend-7b758459b7-vs8sj datahub-frontend Caused by: com.linkedin.r2.RemoteInvocationException: Received error 401 from server for URI <http://datahub-gms:8080/entities/urn:li:corpuser:me@company.com>
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at com.linkedin.restli.internal.client.ExceptionUtil.exceptionForThrowable(ExceptionUtil.java:98)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at com.linkedin.restli.client.RestLiCallbackAdapter.convertError(RestLiCallbackAdapter.java:66)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at com.linkedin.common.callback.CallbackAdapter.onError(CallbackAdapter.java:86)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at com.linkedin.r2.message.timing.TimingCallback.onError(TimingCallback.java:81)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at com.linkedin.r2.transport.common.bridge.client.TransportCallbackAdapter.onResponse(TransportCallbackAdapter.java:47)
...
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at java.base/java.lang.Thread.run(Thread.java:829)
datahub-frontend-7b758459b7-vs8sj datahub-frontend Caused by: com.linkedin.r2.message.rest.RestException: Received error 401 from server for URI <http://datahub-gms:8080/entities/urn:li:corpuser:me@company.com>
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:76)
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	... 4 common frames omitted

Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? Yes button No button

Trying a normal login, sortof the same error, at least it rules out OIDC as the source of the issue

datahub-frontend-7b758459b7-vs8sj datahub-frontend 2023-05-04 14:23:26,011 [application-akka.actor.default-dispatcher-10] WARN  o.e.j.j.spi.PropertyFileLoginModule - Exception starting propertyUserStore /etc/datahub/plugins/frontend/auth/user.props 
datahub-gms-6698965898-bjv4x datahub-gms 2023-05-04 14:23:26,014 [qtp447981768-250] WARN  c.d.a.a.AuthenticatorChain:80 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Provided credentials do not match known system client id & client secret. Check your configuration values...), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Authorization header missing 'Bearer' prefix.)]
datahub-frontend-7b758459b7-vs8sj datahub-frontend 2023-05-04 14:23:26,015 [application-akka.actor.default-dispatcher-10] ERROR p.api.http.DefaultHttpErrorHandler - 
datahub-frontend-7b758459b7-vs8sj datahub-frontend 
datahub-frontend-7b758459b7-vs8sj datahub-frontend ! @81ejojg1e - Internal server error, for (POST) [/logIn] ->
datahub-frontend-7b758459b7-vs8sj datahub-frontend  
datahub-frontend-7b758459b7-vs8sj datahub-frontend play.api.UnexpectedException: Unexpected exception[RuntimeException: Failed to generate session token for user]
datahub-frontend-7b758459b7-vs8sj datahub-frontend 	at play.api.http.HttpErrorHandlerExceptions$.throwableToUsefulException(HttpErrorHandler.scala:358)
...

Thanks for following up @blue-microphone-24514 - we see this pop up occasionally and clearing cache helps sometimes as well

@astonishing-answer-96712 hey Paul, what does clearing ‘cache’ mean? Does that mean clearing web browser cache or clearing out some other cache resources internal to datahub? (tables/indices)