Hi all and happy new year 🙂 New DataHub user here - we've deployed

0.9.3

to a GKE cluster, (with an Istio service mesh) and are noticing some strange behaviour when trying to run CLI ingestion commands. When I configure

DATAHUB_GMS_URL

to point directly to the GMS service (via an ingress, or port forwarding), all works as expected. But when I point the CLI to the frontend service (via an ingress), with suffix

/api/gms

, I get a

401

error.

<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>\n<title>Error 401 Unauthorized to perform this action.</title>\n</head>\n<body><h2>HTTP ERROR 401 Unauthorized to perform this action.</h2>\n<table>\n<tr><th>URI:</th><td>/entities</td></tr>\n<tr><th>STATUS:</th><td>401</td></tr>\n<tr><th>MESSAGE:</th><td>Unauthorized to perform this action.</td></tr>\n<tr><th>SERVLET:</th><td>restliRequestHandler</td></tr>\n</table>\n<hr/><a href="<https://eclipse.org/jetty>">Powered by Jetty:// 9.4.46.v20220331</a><hr/>\n\n</body>\n</html>

This is initially hard to see initially as the HTML response causes the JSON parser to barf (see

cli_utils.py

, L207), but I narrowed it down as only occurring under the above conditions. Note that I am using a token generated by the root

datahub

user, and have set

METADATA_SERVICE_AUTH_ENABLED

to

"true"

for both the frontend, and gms deployments. I can work with things as they stand and communicate with the GMS service directly, but would like to know if this is intended behaviour, as it contradicts the documentation which suggests I should be able to use the frontend proxy

... we will be shifting to the recommendation that folks direct all traffic, whether it's programmatic or not, to the DataHub Frontend Proxy, as routing to Metadata Service endpoints is currently available at the path

/api/gms

Thanks in advance!

@brave-waitress-14748, hi yes, it should work like that. Can you verify your token's validity by running a graphql query using the token. https://datahubproject.io/docs/api/graphql/getting-started#querying-the-graphql-api

Hey @bulky-electrician-72362, thanks for your response 🙂 Yes, the token I am using works fine for graphql and GMS queries fired directly at the

datahub-frontend

service (tested via

kubectl port-forward ...

) , but the same queries don't work when I hit the

datahub-frontend

ingress, which points to that service. Is there a chance the use of istio as the ingress controller is causing the authentication to fail when data arrives at the frontend via the ingress? Also, I've noticed the UI sends graphql queries to

<frontend-ingress>/api/v2/graphql

, the docs refer to

/api/graphql

(no mention of

/v2

). All testing has used`<frontend-service or frontend-ingress>/api/graphql` as per the docs.

Hey, because the queries work against the port-forwarded frontend and not when ingressed, I assume the ingress is the problem. • Are you using the ingress created by our helm chart? • Does istio passed all the headers to service? • Could you check the logs on the frontend pod to see what can be the issue?

because the queries work against the port-forwarded frontend and not when ingressed, I assume the ingress is the problem

Yup 100% agree. I've forwarded your questions to our platform team (the folks who set all this up), but I can confirm right off the bat that the ingress has been modified to use the istio equivalent CRD (a

VirtualService

) and the frontend contains no logs. Is there a way to run the frontend in debug mode so we can see what headers its receiving when the request arrives?

I think if you add

-Dlogback.debug=true

to you JAVA_OPTS env var in the frontend pod it will log on debug level.

no dice unfortunately. I had to edit the

JAVA_OPTS

in the

start.sh

but still no logs. my

JAVA_OPTS

looks like this now:

export JAVA_OPTS='-Xms512m \
-Xmx1024m \
-Dhttp.port=9002 \
-Dconfig.file=datahub-frontend/conf/application.conf \
-Djava.security.auth.login.config=datahub-frontend/conf/jaas.conf \
-Dlogback.configurationFile=datahub-frontend/conf/logback.xml \
-Dlogback.debug=true \
-javaagent:jmx_prometheus_javaagent.jar=4318:/datahub-frontend/client-prometheus-config.yaml \
-Dpidfile.path=/dev/null'

But logs remain as above

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/datahub-frontend/lib/com.google.inject-guice-4.2.2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
SLF4J: Failed to load class "org.slf4j.impl.StaticMDCBinder".
SLF4J: Defaulting to no-operation MDCAdapter implementation.
SLF4J: See <http://www.slf4j.org/codes.html#no_static_mdc_binder> for further details.
ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

Are the ingresses against GMS and frontend identical?

Yep, with the exception of the service and port they're pointing to.

Seems good, Do you use OIDC by any chance or see related error in the frontend?

not using oidc, and yep the UI works fine otherwise, so assume something to do with how the

Authorization

header is interpreted vs. the cookie.

Did you guys setup proper istio authorisation policies for ingress? We are also using istio service mesh and never faced a problem.