Hi team I want to delete the data pipeline and task ingested via airflow I am giving this command but it is not working Datahub delete —env PROD —entity_type data pipeline And also I tried by adding —platform airflow in the end but it is not working

Hi Team, I am deploying datahub in gitlab CI and it is failed with the error

Error: UPGRADE FAILED: error validating "": error validating data: unknown object type "nil" in Secret.data.token_service_salt

it runs fine when I execute manually from my console, this the command that I am using.

helm -n bi-datahub upgrade --install datahub datahub/datahub -f helm/charts/${ENV}/datahub.yaml

Help would be highly appreciated.

Hello, am looking at the helm chart for datahub, was wondering what this encryptionKey is actually encrypting. Thanks in advanced!

Hi everyone, I am deploying a fresh instance of datahub on kubernetes and got it to spin up just fine using the example secrets from the helm chart docs, i.e.

kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub

However, I decided I wanted a more secure password for mysql so I uninstalled datahub & the prerequisites, recreated mysql-secrets with my new password, and even deleted the PVC for good measure (don't worry, there wasn't any data in it 😉). However, when I try to reinstall datahub-prerequisites, the mysql pod throws an error saying

'Access denied for user 'root'@'localhost' (using password: YES)'

. I've tried changing the password in case there was some character mysql didn't like, as well as deleting the PVCs for the other prereq items but no luck. However, when I change the password back to datahub like in the example it works fine again. What is the process supposed to be for changing the mysql password? Is there something else that needs to be updated that I'm missing? Any help would be appreciated. Thanks!

Hello everyone! We are trying to deploy the datahub with PostgreSQL database instead of MySQL. We are consistently getting NPE in the logs of datahub-gms service. Anyone faced similar issue ? can someone help me on this ? I will add the details in the thread.

Can we filter out dbt nodes in dbt recipe itself ?

One thing I have my yaml files on storage grid and I have created configmap having recipe as S3 source but while running I am getting “Java gateway process exited before sending its port number”

Hi team I am written an s3 recipe and added endpoint url But while running I am error As extra field not permitted

Hi team I have dbt manifest generated by airflow and I want to ingest that in datahub So initially we thought to put this manifest file on s3 and use recipe with source type as s3 and read from there and then ingest in datahub but it seems like this is not correct way So can we use this way or there is some other way to automate this process ? And I can stateful ingestion option in dbt recipe When and which case we prefer that ?

Hi team, I can't fetch the created ingestion and secrets from the UI, it was working fine previously. I had tried to change the Mysql instance recently, after that I can't see those data anymore from UI. Here are the errors from UI and GMS, can you please help or give some hint how to solve the problem, many thanks in advance:

Hey All! is there a simple dockercompose to use for datahub without having to set up jq? like the raw yaml itself

Hi there, I am trying to deploy the helm chart on an EKS cluster with all AWS managed services. For the datahub-gms deployment, you need access to the Glue schema registry. Unfortunately you need AWS credentials for this. I want to use least privileged credentials by associating an IAM role to a service account. I configured the IAM role and have this in my values.yml:

datahub-gms:
  enabled: true
  image:
    repository: ${local.ecr_image_prefix}/linkedin/datahub-gms
    tag: ${local.datahub_image_version}
  serviceAccount:
    create: true
    name: "datahub-gms"
    annotations:
      <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: ${module.iam_eks_role.iam_role_arn}

I tested that my service account works correctly by spinning up a test AWS CLI pod (last step in here: https://aws.amazon.com/premiumsupport/knowledge-center/eks-pods-iam-role-service-accounts/) However, my datahub-gms pod is still crash looping and is still assuming my worker role, instead of my least privileged IAM role:

Caused by: software.amazon.awssdk.services.glue.model.AccessDeniedException: User: arn:aws:sts::XXX:assumed-role/mixed-1-node-group-20220530082254040100000001/i-08d04210802a93209 is not authorized to perform: glue:GetSchemaByDefinition on resource: arn:aws:glue:eu-west-3:XXX:registry/neo-datahub-kafka-pro because no identity-based policy allows the glue:GetSchemaByDefinition action (Service: Glue, Status Code: 400, Request ID: XXX)

I am wondering if anyone has encountered this before and if maybe the application doesn’t use the default credentials provider chain?

Hi, we have datahub running on an EKS cluster which is working well. We are currently experiencing an issue where whenever we restart the cluster or perform an upgrade all of the datahub ingestion secrets are wiped and we have to manually re-insert them into the UI. Does anyone know what the issue is here?

Hi, Need some information for deployment with kubernetes. Any guidance on how to estimate the resources required in terms of CPU, memory and disk space with such deployment. Planning to use repository as Postgresql outside the containers to persist the data. Remaining services will be with the containers. Please help with some pointers on how to estimate the resource requirement. Will have few hundreds of datasets to start with and will scale up if needed. Thanks.

Hi all, I’m new to the DataHub world and am trying to deploy v0.8.35 using Helm charts via Terraform. I’m deploying on an EKS cluster, using Confluent Cloud + MySQL + Elasticsearch (for both search & graph). I keep hitting errors on datahub-gms, specifically on readiness & liveness probe failures (details below). I might just be missing something basic, but could someone help me figure out what else I can try? I tried setting the following thresholds but they didn’t seem to change anything:

datahub-gms:
  livenessProbe:
    initialDelaySeconds: 60
    periodSeconds: 30
    failureThreshold: 80
  readinessProbe:
    initialDelaySeconds: 60
    periodSeconds: 30
    failureThreshold: 80

Thanks in advance!

Hi Datahub friends! Has anyone gotten the deployment working on AWS ECS? We’re trying to deploy on ECS (

bridge

network mode) with 1 service and 1 task definition with multiple containers, and currently running into a problem where the containers are not able to communicate with each other via hostname (i.e.

zookeeper:2181

). Since then we’ve tried using

awsvpc

network mode and changed all the hostnames to

localhost

, and while that solved the connectivity problems, there were further issues spinning up

datahub-gms

and overall not sure if this is the right path forward. Please let me know, thank you!

How quick are the new helm charts updated for the 0.36 version?

Hi there 🙂 Anyone succeded to deploy on minikube ? Any version of datahub. Starting minikube like this :

minikube start --memory 16000 --cpus 6 --nodes 3 --driver=docker --force

Still having problem trying to deploy prerequisites :

helm install prerequisites datahub/datahub-prerequisites -n demo

This message was deleted.

hi! I have just updated from 0.8.34 to 0.8.36 and the login is broken

Hello, we have datahub deployed on EKS using the helm chart. Recently we have found that we are no longer able to deploy the prerequisites of datahub because bitnami had removed old versions of Kafka and MySql. With research we found that they have to completely changed their retention policy and deleted everything larger than 6 months old. https://github.com/bitnami/charts/issues/10539 We upgrade kafka and mysql to the newest versions but now we are getting errors when ingesting MySQL and SNOWFLAKE assets. The MySQL errors are simple syntactic errors that are likely occurring due to different versioning. Does any one have a solution deploying the prerequisites with the correct versions now that bitnami has deleted them or do we need to wait for the ingestion jobs to be rewritten to the latest versions of mysql/kafka?

Has anyone deployed DataHub using Rancher?

Hello, Looks like lot of people deployed DataHub on EKS already. I am in the process of doing the same. I am having an issue with

kafka-setup-job

pod when configuring Datahub to work with SSL AWS MSK. Is there an example of how to create

datahub-certs

secret on K8s? I found one for creating mysql and elasticsearch secrets in Datahub docs but not for SSL secrets for Kafka. If anyone could give me a sample snippet of how to create

datahub-certs

secret would help a lot. Thanks in advance!

Hi @early-lamp-41924, Thanks for the pointers given in the above thread. I built a custom image for

kafka-setup-job

to pickup

SASL_SSL

related properties via Helm Chart, I am seeing this error -

[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.                                                                                                                                                             
│ org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient                                                                                                                                                                              
│     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:535)                                                                                                                                                           
│     at org.apache.kafka.clients.admin.Admin.create(Admin.java:75)                                                                                                                                                                                          
│     at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)                                                                                                                                                                              
│     at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:138)                                                                                                                                                                         
│     at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)                                                                                                                                                                     
│ Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule                                                                                 
│     at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172)                                                                                                                                                           
│     at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)                                                                                                                                                                    
│     at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)                                                                                                                                                       
│     at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)                                                                                                                                                                     
│     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)

Any pointers on how to get this Module included to the path/classpath? I see a classpath variable export in the Dockerfile of the

kafka-setup-job

container.

This message was deleted.

Hi team - the standard helm chart (which is great btw) includes ingress for the rest endpoint and web frontend, but not kafka. I'm getting some throughput issues with the rest endpoint for ingestion and so wanted to try using kafka as the sink for ingestion. I tried setting up ingress for Kafka but rapidly got to a stage where I couldn't get things to work. Does anyone have a working setup where the kafka ingestion endpoint is available for use while deployed in a kubernetes cluster? If you do - how did you set things up and how well does it work?

Hello How can i get the list of users that have logged in into datahub in the past X days. @magnificent-notebook-88304

Hi Team. Has anyone tried to deploy DataHub as a multi tenant application where a single deployment stores metadata for multiple tenants? If so, what's the data modelling strategy used to isolate data for different tenants. And is there a way to tie an entity with a tenant?

can yoou tell me how to add extraVolumes to my values.yaml I wanna change default password 🙂