Hello Everyone, I need to make Business Glossary available for my local Datahub instance. The current version is v0.8.34 so Glossary is not here. So the question is if I follow the official updating docs of Datahub (https://datahubproject.io/docs/advanced/no-code-upgrade/) would my data, metadata, lineage, domains, ingestions deleted?

Perhaps more appropriate in this channel

I have a SQLServer source,but no comments. The comments are in a csv file,table head like table-name ,column-name,comment,how can I merge the comments to the mssql source' tables ?

hi team.I would like to know how our system creates es indexes, such as 【datahubstepstateindex_v2】 I did not find any relevant creation logic for the index in the code。

Hi team Can we utilise datahub actions framework that will replicate data from one datahub instance to another ?

Hi, I am trying to deploy Datahub in EKS but the pods are giving me the following error, does anyone know what is wrong?

kubectl get pods

NAME READY STATUS RESTARTS AGE datahub-elasticsearch-setup-job-4zh9g 0/1 Error 0 28m datahub-elasticsearch-setup-job-7d58l 0/1 Error 0 34m datahub-elasticsearch-setup-job-d9698 0/1 Error 0 36m datahub-elasticsearch-setup-job-g58tr 0/1 Error 0 31m datahub-elasticsearch-setup-job-hnj9n 0/1 Error 0 38m datahub-elasticsearch-setup-job-r8r9d 0/1 Error 0 25m datahub-elasticsearch-setup-job-t657v 0/1 Error 0 20m elasticsearch-master-0 0/1 Pending 0 10d elasticsearch-master-1 0/1 Pending 0 10d elasticsearch-master-2 0/1 Pending 0 10d prerequisites-cp-schema-registry-5f89dd4974-sff65 1/2 CrashLoopBackOff 2691 (4m37s ago) 10d prerequisites-kafka-0 0/1 Pending 0 10d prerequisites-mysql-0 0/1 Pending 0 10d prerequisites-zookeeper-0 0/1 Pending 0 10d

Hi everyone, while I'm trying

datahub docker quickstart

, it seems my server can't reach GitHub due to my org's policy. So I'm about to try

datahub docker quickstart --quickstart-compose-file <path to compose file>

. And I can see there are two files from GitHub: 1. docker-compose-without-neo4j.quickstart.yml 2. docker-compose.quickstart.yml When I tried

datahub docker quickstart

, it tried to fetch the first one. If my server already has neo4j on it, then would it be better to use the second one? If I use the first one, then I can't use the graphical lineages?

Hello team ,

1. i have already running kafka, zookeeper ,mysql, schema-registry on my server,  Can these replace into datahub docker yaml file ?

2. can i use postgres RDS instance instead of mysql maria-db ?
please let me know if anyone have any idea.
thanks

Hello. I'm trying to use AWS RDS on DataHub with this page. https://datahubproject.io/docs/deploy/aws/ I already have an Aurora MySQL instance, database, user ID, and password created for Datahub. When I execute the command as shown in the guide, I keep encountering errors. Is there something wrong with my command? this is my command and yaml.

kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=<<password>>

this password is my db password in aws mysql.

sql:
    datasource:
      host: "<<rds-endpoint>>:3306"
      hostForMysqlClient: "<<rds-endpoint>>"
      port: "3306"
      url: "jdbc:mysql://<<rds-endpoint>>:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8"
      driver: "com.mysql.jdbc.Driver"
      username: "root"
      password:
        secretRef: mysql-secrets
        secretKey: mysql-root-password

rds-endpoint is the writer endpoint value of the my AWS MySQL instance that I created. And I changed the username to another userid that I created instead of using "root". Do I have to use only "root" as the username value? Could you please provide me with the correct way to connect AWS RDS to DataHub? Thank you in advance.

hey all, qq - when we see logs like this (top newer than bottom), does that mean the GMS container is waiting for a message on the

DataHubUpgradeHistory_v1

kafka topic before it continues initializing? this is for 0.10.0 to 0.10.1

This message was deleted.

Hello, we have deployed datahubl latest chart in AKS and works fine. we are configuring ingress for datahub-frontend through nginx ingress controller using the below YAML: ingress: enabled: true annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/cors-allow-headers: Content-Type nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, OPTIONS nginx.ingress.kubernetes.io/cors-allow-origin: '_'_ nginx.ingress.kubernetes.io/enable-cors: "true" nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/force-ssl-redirect: "true" nginx.ingress.kubernetes.io/rewrite-target: "/$2" hosts: _- host: x_xx.com paths: - /hub(/|$)(.) # redirectPaths: # - path: /hub(/|$)(.) tls: - secretName: xxx-secret hosts: - xxx.com but the context path is not working as we are getting 200 response but page is not loading. there are two arguments in ingress, PATH and REDIRECT_PATH. can someone suggest how configuring these both?

Hello everyone, I encountered some problems when developing the datahub project. I would like to ask you what system do you use for development, is it mac, linux or windows? And what is your development IDE?

hi,team. when i run command

docker pull acryldata/datahub-upgrade:head && docker run --env-file /Users/work/docker.env acryldata/datahub-upgrade:head -u NoCodeDataMigration

Wrong prompt

ERROR SpringApplication Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'upgradeCli': Unsatisfied dependency expressed through field 'noCodeUpgrade'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'entityRegistryFactory': Unsatisfied dependency expressed through field 'configEntityRegistry'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configEntityRegistry' defined in class path resource [com/linkedin/gms/factory/entityregistry/ConfigEntityRegistryFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: ../../metadata-models/src/main/resources/entity-registry.yml (No such file or directory)

Hi all. We connect to ElasticSearch via credentials which are managed externally and are regularly regenerated, causing us to lose our connection in a running GMS container. Is there a health check in GMS which checks we have a valid, authenticated connection to ElasticSearch? If GMS died when we lost connection to ElasticSearch then k8s would just start it up again with the new credentials. Right now we just have a regular restart which is less than ideal.

We switched our GMS db from postgres 13 to postgres 15 some days ago, and now we see warnings like

unrecognized node type: 380

in the database log. Has anyone seen this before?

Hey all! I currently have one dev and one prod datahub deployment on EKS. The only difference between them is that the prod one has OIDC auth enabled, while the dev one doesn't. The problem I'm facing is with the analytics page. I can see all usage analytics on the dev enviroment, but on the prod one the analytics page is completely empty. Using datahub version 9.6.1 and helm chart version 2.148. Appreciate any help!

hii all, can someone guide me how to authenticate datahub cli ,bcz my datahub is running on eks server

Edit: I didn't change anything, but this is working now. Logs show it's running CLI version 0.10.0.7. --- Original: After updating to v0.10.2, datahub-actions v0.0.12 ingestion is failing due to pip not finding acryl-datahub==0.10.2 - is that still pending? Maybe I have something wrong in my docker-compose. It's for ClickHouse if that matters.

ERROR: Could not find a version that satisfies the requirement acryl-datahub[clickhouse,datahub-kafka,datahub-rest]==0.10.2

Hi Everyone My system-update job is failing with below errors as attached in the screen shot , can any one please assist

client.go:770: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
upgrade.go:436: [debug] warning: Upgrade "datahub" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition

when I ran kubectl describe pod <name of the pod> I see datahub-auth-secrets is not created automatically and throws error Error: secret "datahub-auth-secrets" not found Helm values has provision secrets set to be true

metadata_service_authentication:
      enabled: true
      systemClientId: "__datahub_system"
      systemClientSecret:
        secretRef: "datahub-auth-secrets"
        secretKey: "token_service_signing_key"
      tokenService:
        signingKey:
          secretRef: "datahub-auth-secrets"
          secretKey: "token_service_signing_key"
        salt:
          secretRef: "datahub-auth-secrets"
          secretKey: "token_service_salt"
      # Set to false if you'd like to provide your own auth secrets
      provisionSecrets:
        enabled: true
        autoGenerate: true

Hi team, we are currently working on evaluating Datahub for our internal data cataloging use-cases. I want to pre-emptively gather all the differences between a self-hosted Datahub and a Managed Datahub. This will enable us to decide on the deployment steps early on and align Devops for the same.

Hello team, we have developed based on 0.8.44 and added some extension aspects for customization. We are lagging behind a lot of versions, and I have been trying to upgrade recently, but I have encountered quite a few difficulties. Firstly, I will update based on the datahub-upgrade docker. I downloaded datahub upgrade 0.9.6.1 and 0.8.44, and run them with the following errors. 0.9.6.1->

[com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.lang.IllegalArgumentException: Aspect queryKey does not exist

0.8.44->

[com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.lang.IllegalArgumentException: Aspect embed does not exist

I found that the biggest change is still the index change. I found that the logic for creating the index has been placed in the upgrade-docker. Previously, I remember that the program also created it when it started. Currently, 1.do I have a way to directly create the latest indexes.? and then re-ingest data. 2.whether datahub-upgrade docker is the only channel to init elasticsearch-index?

Hey all, We have currently deployed 0.10.0 version in our environment what we are observing that gms pods are getting restarted many time and we checked the cluster and node logs nothings comes up We checked the splunk logs it’s only showing command exited with error : exit code 143 in every 5 minutes. Is it a bug with this version or what, I am unable to trace it.

Hey everyone! I followed the instructions to change the root user (

datahub

) password found here. While it did update the password, it looks like the user is no longer a super/admin user. Did I miss something here? How do I ensure the

datahub

user is still an admin user?

Hey team, getting liveness and rediness probe failing error in datahub-gms pod, i am deploying 0.10.1 version. Any suggestion?

This message was deleted.

hello Team can someone guide me how to login metabase cli with username and password bcz can’t create token in datahub opensource version

What are the benefits of kubernetes / k8s vs. a simple container for datahub? k8s looks much more scalable but when does the scaling need kick in? We are trying to deploy to 30-100 users, so probably no more than 100 simultaneous users, and realistically, probably no more than 40 at one time. I don't think we'll have metadata for more than 1000 tables * 100 columns.

Starting a new thread to try to move forward on the upgrading issue, started here https://datahubspace.slack.com/archives/CV2UVAPPG/p1678887964885129 Joined office hours today and got some good input from @brainy-tent-14503. I will upgrade the ec2 from t2.large to t2.xlarge tomorrow. I was able to log into it with ssh now, so I can continue with debugging.