Hi everyone, I am seeing this error while running ingestions.

Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 82ms in state APIVERSION_QUERY, 4 identical error(s) suppressed)

My yaml has

springKafkaConfigurationOverrides:
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
    client.dns.lookup: use_all_dns_ips
    basic.auth.credentials.source: USER_INFO

Any ideas? Was just following this doc: https://datahubproject.io/docs/deploy/confluent-cloud/ which is pretty straightforward. Datahub-gms is running fine and the logs say it was able to connect to the cluster.

Hi everyone, currently trying to deploy DataHub to AWS ECS, but im running into an issue where Zookeeper admin server and DataHub GMS are both trying to occupy port 8080, is there some environment variable I’m missing off the Zookeeper task definition that would resolve this issue? Thanks!

Hello folks I'm deploying datahub in AKS following the official documentation and getting below warning. did anyone face this issue earlier or anyone has idea on this please let me know , AKS version : 1.24.6 , I even tried changing the AKS version Warning : (please check the error screenshot) W1219 234355.783521 101257 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget W1219 234357.487684 101257 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget https://datahubproject.io/docs/deploy/kubernetes

Hi everyone, While deploying using helm chart I got the following error:

Error: can't get a valid version for repositories mysql, postgresql, kafka.

I've checked the bitnami repo for the versions used in the prerequisites chart and noticed they are not present so I've used newer versions for these dependencies. Just an FYI if someone gets the same issue and a suggestion to update the versions on these dependencies.

Hi, I have an old Datahub with prerequisites(mysql,kafka,es) in K8S-1. But I want to deploy new Datahub in another K8S-2 with RDS mysql and another kafka and another es because K8S-1 is deprecated and prerequisites are being not manageable, so I want to migrate all the data like column desc, tags to new Datahub. How can I migrate to new one? I am trying to migrate mysql data to new mysql using ‘mysqldump’. So two tables (column_info and mgmt_tables) were moved to new mysql. But they didn’t appear in the new UI.

Hi everyone, I am having an issue with Google SSO. Initially, I thought the error I was receiving related to my Nginx configuration, but after testing locally without a proxy I am seeing the same auth error when trying to sign in. Using the latest Frontend container. This is all I am getting

java.util.concurrent.CompletionException: org.pac4j.core.exception.TechnicalException: Bad token response, error=invalid_grant

My yaml has

- AUTH_OIDC_ENABLED=true
    - AUTH_OIDC_CLIENT_ID=${AUTH_OIDC_CLIENT_ID}
    - AUTH_OIDC_CLIENT_SECRET=${AUTH_OIDC_CLIENT_SECRET}
    - AUTH_OIDC_DISCOVERY_URI=${AUTH_OIDC_DISCOVERY_URI}
    - AUTH_OIDC_BASE_URL=<http://localhost:9002>
    - AUTH_OIDC_SCOPE=${AUTH_OIDC_SCOPE}
    - AUTH_OIDC_USER_NAME_CLAIM=email
    - AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+)

Any ideas on what the problem is or how I can debug further? I did a search and saw a couple others had this issue but never found a way to resolve it.

Hello everyone, My team and I are having a little problem on DataHub deployment with k8s. With the prerequisites/datahub... We've upgraded to k8s v1.25.4, and apparently the chart doesn't support this version (PodDisruptionBudget went from policy/v1beta1 to policy/v1 in v1.25)... Do you know how to deal with that problem? If there is a solution. Thank you in advance.

Hi all - I'm new to datahub .. and have a basic question .. i know that I can connect to databricks with datahub... however can configure great expectations on top of Databricks tables ? Is there a way? Any tutorials or related stuff

I have a question, is there a timeout log out function or where can I set the duration of the session

Hi Team Is there any updates for helm chart repository associated to datahub version v0.9.4? The latest version which i am able to see for helm chart repo is datahub-0.2.120 which is having datahub version v0.9.3 in values.yaml

datahub:
    version: v0.9.3
    gms:
      port: "8080"
      nodePort: "30001"

Im getting this error can any one help me fix this issue?

Hey i am facing issue with datahub kafka setup job it’s showing memory leak error (OOMKILLED) , i am using latest version i.e. 0.9.5 Any solution for this one?

Hello everyone ! I am facing some issues when trying to backup/restore datahub. (It is deployed on K8s, on Azure) I used mysqldump to copy the "datahub" mysql DB. Once the mysql restored from the dump, when I go to the site there are lot of 500 errors everywhere. I tried to restores indices (with this command "kubectl create job --from=cronjob/datahub-datahub-restore-indices-job-template datahub-restore-indices-adhoc"), but it has no effect on the erros. I also tried elasticdump to dump elasticsearch ( the _all index ) and restore it where I restored the mysql and still nothing changed (also tried restoring indices after this step but still no effect on the issue). I am running out of ideas ^^

Hello. I’m trying the AWS deployment guide. By Friday the 23rd, I had completed the Elasticsearch guide. Today on the 26th, when I checked the pods status through the

kubectl get pods

command, the

prerequisites-mysql-0

pod is continuously restarting. So I redeployed the

prerequistites

repo. However, the pod is still running again. Attach log image of

prerequisites-mysql-0

. What should I do?

Hi, I'm using v0.9.2 and would like to upgrade v0.9.5. I got error like this.

Hello hello 🙂 I want to expose the endpoint with my elb, as describe in your doc I already have an EKS cluster, datahub NS & elb endpoint for the frontend I’m not sure what I need to revoke from the steps from the doc Can someone assist pls.?

Hi, I'm using v0.9.2 and would like to upgrade v0.9.5. I got error like this. How can I start ebeanServer? My docker.env is as following.

# Required Environment Variables
EBEAN_DATASOURCE_USERNAME=datahub
EBEAN_DATASOURCE_PASSWORD=datahub
EBEAN_DATASOURCE_HOST=de-test-3:3306
EBEAN_DATASOURCE_URL=jdbc:<mysql://de-test-3:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8>
EBEAN_DATASOURCE_DRIVER=com.mysql.jdbc.Driver

KAFKA_BOOTSTRAP_SERVER=de-test-3:9092
KAFKA_SCHEMAREGISTRY_URL=<http://de-test-3:8081>

ELASTICSEARCH_HOST=de-test-3
ELASTICSEARCH_PORT=9200

DATAHUB_GMS_HOST=de-test-3
DATAHUB_GMS_PORT=8080

@handsome-solstice-96967 @fast-oyster-93603 hey there -- just wondering about the rationale behind removing

master

deployment from the Helm chart in this PR: https://github.com/acryldata/datahub-helm/pull/224

Hello everyone ! I am trying upgrade one of my datahub instances from 0.8.35 to 0.8.45 but the upgrade fail. It is deployed on Kubernetes (Azure) with the official helm charts. The command I use is helm upgrade with the latest chart version i found for datahub 0.8.45 ( => 0.2.108). • "helm upgrade datahub datahub/datahub --version 0.2.108 -n my_namespace" The upgrade start by running a datahub-elasticsearch-setup-job but that job fail (logs below). The job is automatically started again multiple time and fail everytimes witht the same logs until it reach the max retry limit. I already went through the slack's channel about the migration for this range of version (the one archived, that is linked in the doc) but didn't found a solution ^^.

2022/12/28 08:41:12 Waiting for: <http://elasticsearch-master:9200>
2022/12/28 08:41:12 Received 200 from <http://elasticsearch-master:9200>
Going to use protocol: http
Going to use default elastic headers
Create datahub_usage_event if needed against Elasticsearch at elasticsearch-master:9200
Going to use index prefix::
curl: option -k <http://elasticsearch-master:9200/_ilm/policy/datahub_usage_event_policy>: is unknown
curl: try 'curl --help' or 'curl --manual' for more information
/create-indices.sh: line 41: [: -eq: unary operator expected
/create-indices.sh: line 45: [: -eq: unary operator expected
/create-indices.sh: line 47: [: -eq: unary operator expected
2022/12/28 08:41:12 Command exited with error: exit status 1
Policy GET response code is 
Got response code  while creating policy so exiting.

Hi! I’m running an EKS setup for DataHub and when deploying it with

helm install datahub datahub/datahub --values <my_values>.yaml

it creates ELB and S*ecurity Group* (Description: ”_Security group for Kubernetes ELB <name_id> (default/datahub-datahub-gms)_”) which allows traffic from 0.0.0.0/0 to ports 8080 and 4318. How can I limit the sources in the security group inbound rules so that it doesn’t allow 0.0.0.0/0, but only limited sources, or is it necessary to have it with 0.0.0.0/0 sources?

Hi guys, I am trying to get the authentication working with Azure, and in theory I got everything up and running. However, I was trying to get a colleague to login as well and it didn't work because the Redirect URI was as follows: http://localhost:9002/callback/oidc. as instructed in the docs here: https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react-azure. Therefore, I deduced that I would need to use this url instead https://your-datahub-domain.com/callback/oidc. However, in order for this to work I guess I would have to deploy datahub, or set up and host a website? Could somebody point in the right direction of what to do, it would be very appreciated!

Hey team, I've deployed the 0.2.122 helm charts located here (https://github.com/acryldata/datahub-helm/releases/tag/datahub-0.2.122) The deployment appears to be successful... however I am not seeing any of the expected directories in the deployed pods. i.e. I don't see datahub-gms:

/tmp/datahub/logs/gms or datahub-frontend: /tmp/datahub/logs/datahub-frontend

The deployed version is 0.9.3

Any help would be appreciated.

Hi Team, I'm trying to connect to Azure mysql flexible server using below connection string

im wondering if you guys would consider tagging the actions container as part of the release, just to cleanly demarcate which image is for which release

Hi all, I'm done following the deployment of datahub via Helm Chart. However, I'm facing some problems related to connection to source system. My company has a policy that all servers cannot talk to any other server until a Access Control Request has been made. There are several solutions that I could think of, but not sure if it is the correct thing to do. • Request Access control between all k8s servers to all source system. The will be a pain when new server are added and when the number of cluster grew.

Hi all, I'm done following the deployment of datahub via Helm Chart. However, I'm facing some problems related to connection to source system. My company has a policy that all servers cannot talk to any other server until a Access Control Request has been made. There are several solutions that I could think of, but not sure if it is the correct thing to do. • Request Access control between all k8s servers to all source system. The will be a pain when new server are added and when the number of cluster grew. • Only Request Access control to some of the k8s servers and label those servers as "talker" and only deploy pods that are responsible for query metadata to those talker server. In this case what is the pod that does the querying? I assume it Datahub GMS. Has anyone faced this problem before and how did you solve it?

Hi guys, Can somebody help me with how to set up ingress to make the front-end in the kubernetes cluster publicly accessible? How I'm trying to configure it now is by changing the values.yaml file and reupgrading the helm charts for datahub/datahub charts. I first thought that I had to create my own ingress resource and ingress controller, however I see that when I helm upgrade with the ingress: enabled: true that it is creating an ingress resource. Below I will share the values.yaml file that I'm using. In any case, can somebody guide me step by step how to set up the ingress with the front-end? the documentation just says to expose it but doesn't really explain how.

datahub-frontend: enabled: true image: repository: linkedin/datahub-frontend-react # tag: "v0.9.3" # # defaults to .global.datahub.version # Set up ingress to expose react front-end ingress: enabled: true hosts: - host: my-host-name paths: - path: / pathType: Prefix backend: service: name: datahub-datahub-frontend port: number: 9002 redirectPaths: []

Hi there, We successfully deploy DataHub and all of its prerequisites. There is just one problem we discovered when browsing the logs. Unfortunately, the elastic index for the usage events is not created by the elasticsearch-setup-job (which we use in version

0.9.3

). Consequently, we only get an empty analytics page shown in the UI. We could fix this by running

curl -XPUT '<http://localhost:9200/_data_stream/datahub_usage_event>'

manually. Is this a known issue? Has anyone an idea how to integrate this command in the datahub deployment so that we don't need to execute it manually for every deployment?

I think @incalculable-ocean-74010 mentioned in the past that datahub was not been tested with SSL connection between GMS and frontend before, but can I check if anyone has done so successfully in k8s?