One more question, where I can find which dependencies are installed along with

acryl-datahub[airflow,great-expectations]==0.8.31.5

package?

I have one more interesting issue that I’m trying to resolve now. When I added great_expectations action to ingest validation data to DataHub, my task failed with memory leak.

- name: datahub_action
    action:
      module_name: datahub.integrations.great_expectations.action
      class_name: DataHubValidationAction
      server_url: <http://datahub-gms:8080>

Do you have any idea what can be the problem?

Hi all - looking for some help regarding OIDC using Keycloak! I have managed to setup datahub to run on minikube locally Keycloak is also running via minikube and is being exposed with ingress. In addition, I also followed these instructions: https://minikube.sigs.k8s.io/docs/tutorials/custom_cert_ingress/ to enable TLS for that ingress. To enable OIDC with keycloak for datahub I added the following to the values.yaml

extraEnvs:
    - name: AUTH_OIDC_ENABLED
      value: "true"
    - name: AUTH_OIDC_CLIENT_ID
      value: datahub-frontend
    - name: AUTH_OIDC_CLIENT_SECRET
      value: <secret>
    - name: AUTH_OIDC_DISCOVERY_URI
      value: <https://keycloak.192.168.49.2.nip.io/realms/datahub/.well-known/openid-configuration> 
    - name: AUTH_OIDC_BASE_URL
      value: <http://192.168.49.1:9002>

Then ran :

helm install datahub datahub/datahub --values values.yaml

However, when accessing http://192.168.49.1:9002 I get the message "Oops an error occurred" on the front end. The datahub-frontend pod logs show the following:

Unexpected exception[CompletionException: org.pac4j.core.exception.TechnicalException: javax.net.ssl.SSLHandshakeException: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I note that @handsome-football-66174 had an similar error which was due to a problem with the java server not being able to trust the SSL certificate. I have tried to following the suggestions from @big-carpet-38439 and@orange-night-91387 on the thread here https://datahubspace.slack.com/archives/C029A3M079U/p1644017149655469?thread_ts=1643841330.686689&cid=C029A3M079U. To create the certificate I used mkcert and ran the following commands 1) mkcert -install to create a local CA certificate and add it to the Java truststore 2) mkcert -key-file key.pem -cert-file cert.pem keycloak.192.168.49.2.nip.io *.keycloak.192.168.49.2.nip.io to create a locally trusted certificate I then created a ConfigMap and copied the contents of the truststore (cacerts file) and added the JAVA_OPTS, extraVolumes and extraVolumeMounts similar to here: https://datahubspace.slack.com/archives/C029A3M079U/p1644017149655469?thread_ts=1643841330.686689&cid=C029A3M079U Any help would be much appreciated! Thanks.

Hello! I recently made a post in the UI channel about updating themes for datahub running in Kubernetes. Apparently, making these changes for Kubernetes deployments requires rebuilding the datahub-frontend container. This is what I originally suspected and have already made some efforts to rebuild the image, but have run into some trouble with gradle. I'm running the following command to build the

datahub-frontend

image:

DOCKER_BUILDKIT=1 docker build -t "$IMAGE_TAG" -f ./docker/datahub-frontend/Dockerfile

However, doing so eventually gives me the following error:

* What went wrong:
Execution failed for task ':li-utils:generateDataTemplate'.
> Process 'command '/usr/lib/jvm/java-1.8-openjdk/bin/java'' finished with non-zero exit value 1

As well as this further down in the stack trace:

> Task :datahub-frontend:compilePlayBinaryPlayRoutes

I haven't been able to diagnose this issue, at first glace it seems like java issue, perhaps something with the version. But I am confused as to why an error would occur in the first place as we have not made an substantive changes to the code from the source repo and the image should be built in a blank container. Now I originally dealt with a series of errors about gradle being unable to download plugins and dependencies, so I had to make a few changes to allow these to be pulled in our network environment. I will list those below: • Updated the

distributionUrl

in

gradle-wrapper.properties

to point to our Nexus repo • Added a

pluginManagement.repositories.maven

object with our nexus url and credentials to

settings.gradle

• Added a new repository object for our nexus repo to

repositories.gradle

and commented out all other repos (as they would be blocked when accessed directly and are configured to be proxied by our nexus repo) • Added a few lines to the datahub-frontend

Dockerfile

to accept nexus credentials as ARGs, set them as ENV variables, and used them to set

-Dgradle.wrapperUser

&

-Dgradle.wrapperPassword

in the

GRADLE_OPTS

ENV and set new command line arguments

-PsonatypeNexusUsername

&

-PsonatypeNexusPassword

to the

./gradlew :datahub-frontend:dist

command. Has anyone encountered a build issue like this before or is able to assist in the trouble shooting? I appreciate it and have a great week!

Hi, I have a question regarding the datahub storage. So let's say I had uploaded the dataset to the datahub and it is visible on main page of the datahub, where is the dataset file stored locally in datahub folder? I would like to make backup for them and I am not sure where to find them now

Hi guys, I´m trying to install Datahub on AWS but I´m getting the following message when I run the command: "helm install datahub datahub/datahub" Does anyone know how to fix it? Thanks.

While trying to replicate the GreatExpecations example shown in the townhall i am getting below error.

great_expectations --v3-api checkpoint run my_checkpoint
Using v3 (Batch Request) API
No module named `datahub.integrations.great_expectations.action` could be found in your plugins directory.
    - Please verify your plugins directory is configured correctly.
    - Please verify you have a module named `datahub.integrations.great_expectations.action` in your plugins directory.

Hi team, How can I set a user created via OIDC as root user like 'datahub' ? I'd like to grant all policies(Root User - *) to the user.

Hi, I ran "datahub docker quickstart" as it's said in: https://datahubproject.io/docs/quickstart/, unfortunately I am seeing an error saying "ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp3s1qc55a.yml'". I have the latest version of datahub CLI 0.8.31.6. Is there any better way to lauch datahub? I'll be putting the full error message in the comments. Thank you all.

Hi guys, i have a problem in metabase ingestion it works successfully but I can't find it in datahub Slack Conversation

hey all https://datahubspace.slack.com/archives/CV2KB471C/p1648733484307099?thread_ts=1648732883.652859&cid=CV2KB471C

Hey folks! I'm trying to delete a container information from my datahub using datahub cli. The action performs well but when i access the datahub itself the informations still there? What am i doing wrong? Code that i used:

datahub delete --urn urn:li:container:37c7bb069d2b23708574c0bdf835dea8 --entity_type container --hard

The information in the website after the command is in the print bellow.

I ran "datahub docker quickstart" and it keeps restarting the mysql-setup, elasticsearch-setup and kafka-setup. Also the datahub-gms is unhealthy, I can acess the UI login page but I can't login.

datahub docker check

says "datahub-gms is running but not healthy" Any thoughts?

I've seen that some of my running DAGs from Airflow are not getting imported in Datahub. It looks like this error is caused by when the DAGs is configured with a template of the Param Class ;I don't know exactly why this error occours, but it could be relate to type of params, which is a dictionary type. Issue:https://github.com/datahub-project/datahub/issues/4546

Hey folks, i'm trying to ingest some data from my redshift datasource but it's running for like 5 hours (and we have about 24 tables). Seems like that the ingestion finished but the status is wrong in the ingestion section. What can i do to help you to debug this?

images are broken on the docs guys.

https://github.com/datahub-project/datahub/issues/4550 Can anyone here help me out with this issue?

I tried pulling the latest version of the code (not the releases) off the repo to check out view ACL, but it seems like the search bar is broken after ingesting some datasets checking mySQL and ES, both datasources have the new dataset records already. GMS is not returning any errors in logs graphql search also not returning me any results after i ingested the datasets. * returns me the results BEFORE ingestion took place.

Guys, when i configure the Ingress on K8s, the frontend turn to this gray page instead of the another one. Then i log in, it redirects to a blank page. Can you help me with this?

I tried to run DataHub using CLI command ‘datahub docker quickstart’, but I got the error like bellow: elasticsearch is up-to-date mysql is up-to-date zookeeper is up-to-date neo4j is up-to-date Starting elasticsearch-setup ... Starting mysql-setup ... datahub-gms is up-to-date Starting elasticsearch-setup ... done Starting mysql-setup ... done Starting broker ... done Starting schema-registry ... done kafka-setup is up-to-date ............. Unable to run quickstart - the following issues were detected: • kafka-setup is still running • datahub-gms is still starting • broker is not running If you think something went wrong, please file an issue at https://github.com/datahub-project/datahub/issues or send a message in our Slack https://slack.datahubproject.io/ Be sure to attach the logs from /tmp/tmppcedmu0q.log How do I figure it out? Please, help me….

Hello, does datahub repository provide some json validator? Let's say that I had created some json files on my own and I'd like to upload them to datahub in two different ways - using the curl command and using file as a source in yml file. I know at this point that the structure for two of them differs and they are not alike with each other, even though they carry the same information to datahub. I am asking this question, because datahub (which I did setup for locally on my server) starts to throw status 500 after some period of time and sometimes there is error visible in the logs stating that the json inside MySql database is erroneous, despite of ingesting it beforehand to datahub.

Hi team, We faced a strange behaviour when we search some terms and columns. If we search "daily gmv" then search brings daily gmv and gmv related terms and columns. But when we search "monthly gmv" then nothing returns. Is it a normal situation or is there way we configure search to bring first exact match and then similar assets.

hI all, I'm suddenly facing issues with clean quickstart on CentOS Linux release 7.9.2009 (Core) machine....had no issues with previous versions whatsoever

Unable to run quickstart - the following issues were detected:
- datahub-gms is running but not healthy

Hi we're having trouble enabling OIDC Auth we're using version 2.62 of the helm chart and we've done the suggested config from this issue and we've also tried without quoting it. We're seeing this on the frontend pod --

Caused by: com.typesafe.config.ConfigException$WrongType: env variables: auth.oidc.enabled has type STRING rather than BOOLEAN

This is happening for any variation of the OAUTH_OIDC_ENABLED env var. The YAML currently looks like this:

extraEnvs:
      - name: "AUTH_OIDC_ENABLED"
        value: "true" (have also tried just plain true)
      - name: "AUTH_OIDC_CLIENT_ID"
        value: "SNIP"
      - name: "AUTH_OIDC_DISCOVERY_URI"
        value: "<https://accounts.google.com/.well-known/openid-configuration>"
      - name: "AUTH_OIDC_BASE_URL"
        value: "SNIP"
      - name: "AUTH_OIDC_SCOPE"
        value: "openid profile email"
      - name: "AUTH_OIDC_USER_NAME_CLAIM"
        value: "email"
      - name: "AUTH_OIDC_USER_NAME_CLAIM_REGEX"
        value: "([^@]+)"
      - name: "AUTH_OIDC_CLIENT_SECRET"
        value: "SNIP"

Anyone?

Hi all, I installed DataHub with

datahub docker quickstart (without source code),

I want to create a new user to login (with username and password). How can I do it ? where can i find user.props file ?

Hi all when upgraded from 0.8.31.5 -> 0.8.31.6 in

datahub-upgrade-job

***************************                                                                                                                                                                                                                             
APPLICATION FAILED TO START                                                                                                                                                                                                                             
***************************                                                                                                                                                                                                                             
                                                                                                                                                                                                                                                        
Description:                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                        
The bean 'kafkaProducerFactory', defined in class path resource [org/springframework/boot/autoconfigure/kafka/KafkaAutoConfiguration.class], could not be registered. A bean with that name has already been defined in URL [jar:file:/datahub/datahub-u
pgrade/bin/datahub-upgrade.jar!/BOOT-INF/lib/factories.jar!/com/linkedin/gms/factory/kafka/KafkaProducerFactory.class] and overriding is disabled.                                                                                                      
                                                                                                                                                                                                                                                        
Action:                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                        
Consider renaming one of the beans or enabling overriding by setting spring.main.allow-bean-definition-overriding=true

hello , i have a problem when trying ingest MySQL

TypeError: 21:30:00 is of type timedelta which cannot be serialized

does anyone has experience how to solve it ?

Hi, is there any way to use S3 as a source for datahub, but with my own endpoint that is not present on AWS?

Hello all, we have been working on testing the data hub deployment on AWS EKS. All the services seem running well and we have configured the ingress using an existing certificate and till this point all works well. However when we open the frontend and login using the default user (

datahub

) we get an empty page and the following error:

react-dom.production.min.js:216 Error: Could not fetch logged in user from cache. + Exception while fetching data (/corpUser) : java.lang.RuntimeException: Failed to retrieve entities of type CorpUser

useGetAuthenticatedUser.tsx Uncaught (in promise) Error: Could not fetch logged in user from cache. + Exception while fetching data (/corpUser) : java.lang.RuntimeException: Failed to retrieve entities of type CorpUser.

Also this in the pods logs:

Caused by: com.linkedin.r2.message.rest.RestException: Received error 404 from server for URI <http://datahub-datahub-gms:8080/dataJobs>

As we are following the recommended procedure, would like to know if you have seen these issues before.