Hej Team, I am trying to explore DataHub and started with using quickstart. I have encountered the following error

Unable to run quickstart - the following issues were detected:

- kafka-setup is still running

- datahub-gms is running but not healthy

The message is slightly misleading - kafka-setup is not running. Anyway - I am looking at an earlier post - https://datahubspace.slack.com/archives/C029A3M079U/p1650276230288759 where the user tries an older version to get it going (v0.8.33). I am apparently using

DataHub CLI version: 0.8.45

Is it still necessary to try an older version? The datahub-gms container logs is attached if you are interested to see the error.

Hi All, We are trying to upgrade datahub to v0.8.45 in our kubernetes cluster and observed "datahub-elasticsearch-setup-job" pod is getting failed and because of that frontend and gms pods also not deploying . Need your suggestion / solution on this issue . Below is the error logs that we observed .

Hello! I'd like to install Datahub on my K8s cluster using helm (https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites). But the problem is my K8S cluster has no internet access. So i need to install al the docker images that this Datahub Helm Chart uses. I got 2 questions. 1. Which images are going to be used for helm install locally ? Because I see two different tags for the same image. For example "acryldata/datahub-actions", v0.0.1 for subcharts, v0.0.7 for another (Subcharts https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/acryl-datahub-actions/values.yaml), ( https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml). 2. When you go to prerequsite(https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites), I can't find any informations(more specifically repository:tag info) about the images. How do i figure out the exact repository:tag informations of dependency images that are going to be used? So that i can install Datahub on K8S locally(without internet)! I would really appreciate it if you answer my questions.. thanks for your time!

Hi, all. I want to use datahub with ElasticSearch manadged service and I need to us certificate. My command has format like

curl --cacert CA.pem "https://<user>:<password>@<host>:9200/"

How have I specified path to certificate for gms and elasticsetup job?

Hey guys! I am trying to have Datahub work without a user at all. I've seen ways to add users, but I do not know if you could just deactivate the default user all together!

Hey everyone, I seem to be getting an error when running

datahub docker quickstart

; it's returning

Command '['docker', 'compose', '-f', '/home/stephane/.datahub/quickstart/docker-compose.yml', '-p', 'datahub', 'logs']' returned non-zero exit status 125.

Has anyone ever encountered this?

Hi, I'm attempting to deploy datahub via minikube using M1 machine. I never had issue with intel chip before. But this is what I get when trying to `helm install prerequisites datahub/datahub-prerequisites -n datahub`:

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: resource mapping not found for name: "elasticsearch-master-pdb" namespace: "" from "": no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
ensure CRDs are installed first

Any idea how to fix this. I followed this https://datahubproject.io/docs/deploy/kubernetes/

Hi all, I have troubles using the right AWS role for the datahub actions deployment (acryldata/datahub-actions:v0.0.8) in EKS. I have annotated my serviceaccount with the right role to assume and the pod is using that sa. This role has the permissions to e.g. read from Glue tables. However, when I run an ingest using datahub actions, it is failing with the following message:

ClientError: An error occurred (AccessDenied) when calling the AssumeRoleWithWebIdentity operation: Not authorized to perform sts:AssumeRoleWithWebIdentity

From the debug logs I can see that it has already assumed the correct role:

...
    self = <botocore.client.STS object at 0x7f0f4f853190>
     operation_name = 'AssumeRoleWithWebIdentity'
     api_params = {'RoleArn': 'arn:aws:iam::XXXXXXXXXX:role/ttgsl-test-iam-edp-datahub-gms',
                   'RoleSessionName': 'botocore-session-1664525045',
...

Why does it try to assume a different role when it can use the attached role? What am I missing? Thanks!

Hey waving from afar left Managed to successfully build

v0.8.45

but when running

docker/dev.sh

a

datahub-frontend-react

error arises:

datahub-frontend-react    | ! @7p67np407 - Internal server error, for (GET) [/admin] ->
datahub-frontend-react    |
datahub-frontend-react    | play.api.UnexpectedException: Unexpected exception[ProvisionException: Unable to provision, see the following errors:
datahub-frontend-react    |
datahub-frontend-react    | 1) Error injecting constructor, org.apache.kafka.common.KafkaException: Failed to construct kafka producer
datahub-frontend-react    |   at controllers.TrackingController.<init>(TrackingController.java:47)
datahub-frontend-react    |   while locating controllers.TrackingController
datahub-frontend-react    |     for the 7th parameter of router.Routes.<init>(Routes.scala:45)
datahub-frontend-react    |   while locating router.Routes
datahub-frontend-react    |   while locating play.api.inject.RoutesProvider
datahub-frontend-react    |   while locating play.api.routing.Router
datahub-frontend-react    |
datahub-frontend-react    | 1 error]

Am I missing something? 😕 Thanks in advance

Hej Team! I am trying to get the metadata ingestion going from one of our Oracle DBs. The configuration appears to be straight forward - I have verified the source connections. The ingestion fails though. I have a quickstart setup running v0.8.33. I suspect the problem is with the sink (datahub-rest) because there is a 404 response from localhost:8080. And I have verified the datahub-gms container - it is healthy and running fine (even restarted it). I have attached the error log from the ingestion and also the error message when hitting localhost:8080 I have looked at the troubleshooting tips from https://datahubproject.io/docs/ui-ingestion - but it doesn’t appear to help my case - can you perhaps point me in the right direction?

Hello everyone! I just want to share one thing that a noted and could be crucial. After upgrading my DataHub from 0.8.41 to 0.8.44, the user experience was drastically affected. After a few days of trying to debug, I decided to rollback my development instance to 0.8.41 version. Below, I will share the results of the same query, the first in the production environmt with the version 0.8.44 and the second in the development environment with the version 0.8.41. Searching for "order" - version 0.8.41 First try: | 43 requests | 5.6 MB transferred | 5.6 MB resources | Finish: 5.50 s | DOMContentLoaded: 3.82 s | Load: 4.23 s | Average time is 6 seconds to load Searching for "order" - version 0.8.44 First try: | 32 requests | 5.7 MB transferred | 5.7 MB resources | Finish: 34.64 s | DOMContentLoaded: 3.84 s | Load: 4.36 s | Average time is 14 seconds to load Disclaimer: This performance impact has been reported by ALL users of the platform but the above data does not necessarily prove it. I just want to know if anyone else has gone through the same situation. Does anyone know why?

Hey everyone, I’ve having some trouble with OIDC via google. I’m getting

Failed to resolve user name claim from profile provided by Identity Provider. Missing attribute. Attribute: 'preferred_username'

error. The profile returned from google is missing the

preferred_username

and

username

as well, what do I need to do in order to add it to the profile sent back from google? Is there a workaround? These are the fields I’m getting from the profile provider:

{
at_hash=***, 
sub=112764038129102833931, 
email_verified=true, 
id_token=***, 
iss=<https://accounts.google.com>, 
given_name=***, 
locale=en, 
picture=***, 
token_expiration_advance=-1, 
aud=[***.<http://apps.googleusercontent.com|apps.googleusercontent.com>], 
azp=***.<http://apps.googleusercontent.com|apps.googleusercontent.com>, 
name=*** ***, 
hd=***.com, 
exp=***, 
family_name=***, 
iat=***, 
email=***@***.com
}

With

AUTH_OIDC_USER_NAME_CLAIM=email

I’m getting

Failed to perform post authentication steps. Error message: Failed to provision user with urn

Hello everyone, I wanted to ask a quick question regarding datahub backup. Is there any way to trigger the backup using the cli and specifying the host where datahub is located? Let's say I have server A with datahub running via the docker compose file with some IP On server B I had performed pip install acryl-datahub so it means I have access to the datahub cli. So is it somehow possible to perform the backing up of datahub on server A using cli on server B?

Hello guys, I've set an ingestion pipeline

snowflake-beta

with stateful ingestion and it was working just fine. Then a new version was released and

snowflake-beta

doesn't exist anymore, it was replaced by

snowflake

. However, all my statefull ingestion set with

snowflake-beta

doesn't work anymore. I've now old tables that were already dropped from snowflake but which remains on the Catalog after running an ingestion with

snowflake

and

stateful ingestion enabled

. Is there a way to solve this problem ? Thanks !

Hi! I've been debugging my dataHub deployment on GKE for a few days, and I noticed that most (if not all) of the GMS queries are going straight to the database. I suppose some of them should be done in elasticsearch. Does anyone have an idea what it could be?

Hi Team, Is there any way to automate the tags on the data classification in datahub like Sensitive, Restricted and General. If anyone is aware on this could you please guide me?

Hi guys, I've tried to deploy on azure kubernets, prerequisites are ok but datahub says Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition Looking at the job I found Create datahub_usage_event if needed against Elasticsearch at elasticsearch-master:9200 Going to use index prefix:: curl: option -k http://elasticsearch-master:9200/_ilm/policy/datahub_usage_event_policy: is unknown curl: try 'curl --help' or 'curl --manual' for more information Policy GET response code is Got response code while creating policy so exiting. /create-indices.sh: line 41: [: -eq: unary operator expected

Hi There, Curious if Datahub supports Liquid Variable logic in LookML where an explore can be derived from one of 2 underlying tables?

hey all, is it possible to supply our own "policies.json" file? I see that in the code, the path to the file is hardcoded to be inside the deployment JAR, so i'm not sure https://github.com/datahub-project/datahub/blob/bfb903cfb874db2c19a53e0eb2d3c1c697[…]n/java/com/linkedin/metadata/boot/steps/IngestPoliciesStep.java

Hi Team, Any help is appreciated. We have enable OKTA OIDC multifactor authentication but ui is repeatedly asking for 2nd authentication.

ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature

how to resolve this issue? Thanks in advance

Hi Team, I’m using datahub v0.8.40 and we are using es as graph impl but our datahub is super slow and i found out that this is because graphiql is responding super slow. but when i query to es directly it responds super fast what can make graphiql response so slow? it gets timeout when we make query like searchAcrossEntities Any help or tip would help. I cannot find where to start.

hi here 😀 I could not find any reference on the process to login via ldap configured in jaas.conf these are the things i've tried. I removed the user.prop file from the frontend image(v0.2.87) and added the jaas.conf file. jaas.conf

WHZ-Authentication {
    com.sun.security.auth.module.LdapLoginModule required
    userProvider="ldap://{ldap_host}:{ldap_port}"
    authIdentity="cn=${0},ou=members,o=identitymaster"
    java.naming.security.authentication="simple"
    debug="true"
};

Then there was a problem that I could connect all ID and password I entered. I want to allow access only to IDs with ldap privileges. Any help would be appreciated. Thanks

Hello 🙂 I was trying to ingest metadata from Snowflake using the version 0.8.45.1. It took me more than five hours (instead of ~10 mins with the 0.8.44.4 version) to run and at the end I got this error:

snowflake.connector.errors.ProgrammingError: 390114 (08001): Authentication token has expired.  The user must authenticate again.

Any idea on what's going on and how to prevent that? Thanks

Getting this error in `datahub-gms`:

Error creating bean with name 'ebeanServer' defined in com.linkedin.gms.factory.entity.EbeanServerFactory

I am running MySQL and

mysql-setup

ran without issues so seems connection is fine from that operation. Using same credentials for

datahub-gms

. Anyone have ideas?

Hello! I was trying to ingest data from Snowflake with profiling enabled on version 0.8.44.4 and I believe it broke something during the computation time. My lineage wasn't complete, however when I don't use the SQL Profiling, I have the right lineage (I have the ignore_start_time_lineage boolean set to true for testing purposes). Do you know if I can run lineage and profiling on two separate metadata ingestion runs?

hi everyone, I was trying to

helm install

the datahub applications with a self-managed aws msk instead of the shipped kafka image. But I come across an error while running the kafka setup job. May I have your help in it ? thanks in advance error log:

Error while executing config command with args '--command-config /tmp/connection.properties --bootstrap-server <my-msk-broker-endpoints> --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support INCREMENTAL_ALTER_CONFIGS
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
	at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:335)
	at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:302)
	at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
	at kafka.admin.ConfigCommand.main(ConfigCommand.scala)

spec: Kafka version: 2.2.1 job setting:

kafkaSetupJob:
  enabled: true
  image:
    repository: linkedin/datahub-kafka-setup
    tag: "v0.8.44"
  podSecurityContext:
    fsGroup: 1000
  securityContext:
    runAsUser: 1000
  podAnnotations: {}

This message was deleted.

Hi, I am facing this error while I try to run my Metabase ingestion:

raise HTTPError(http_error_msg, response=self)\n'
           'requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url:

The user/password I saved in the recipe/secrets is the same as I use to login in our Metabase, so, I shouldn’t be ‘unauthorized’. =/ Does someone know what this can be?

hey guys, I'm having issues with Fields tab on chart entity UI. It seems like the browser cannot render the page at all once I click on it....anyone has any clues?

This message was deleted.