Hi, I am deploying datahub v0.8.41 in docker 20.10.17 and have found one thing that I don't know if it is intended to be this way but it doesn't make much sense as far as I know. The thing is that in the 3 basic services needed for the deployment (gms, frontend and actions) the user used to log in is not the same. For example: in datahub-gms is

uid=101(datahub) gid=101(datahub) groups=101(datahub)

but in the datahub-front-end is

uid=100(datahub) gid=101(datahub) groups=101(datahub)

. Is this done on purpose or is it just a mistake?? Thanks in advance for the help!

Hi team We are working on vulnerability for datahub -pre chart We upgraded the elastic search to 8.3.2 version and initially we are using 7.16.2 version But after upgrading the image the count of vulnerabilities are reduced but few Java based vulnerabilities are there We have checked with elastic search community for their resolution and they are saying they are releasing new images version but datahub is not able to accommodate the latest image So can you please let us know on this ?

Hello everyone, We have deployed Datahub v0.8.41 using Kubernetes and we would like to customize the default datahub user login credentials. We found the information related linked here. However, we were wondering if there is any documentation about how to proceed with it in Kubernetes. Thank you in advance!

Hi All! having issue with

datahub-gms

, node (readiness & liveness probe failures). can anyone please share link to helm chart values file that runs

datahub

successfully using AWS managed services for the storage.

17:51:58.700 [main] WARN  o.s.w.c.s.XmlWebApplicationContext:591 - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'siblingGraphServiceFactory': Unsatisfied dependency expressed through field '_entityService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'entityAspectDao' defined in com.linkedin.gms.factory.entity.EntityAspectDaoFactory: Unsatisfied dependency expressed through method 'createEbeanInstance' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanServer' defined in com.linkedin.gms.factory.entity.EbeanServerFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.ebean.EbeanServer]: Factory method 'createServer' threw exception; nested exception is java.lang.NullPointerException
Jul 19, 2022 5:51:58 PM org.neo4j.driver.internal.logging.JULogger info
INFO: Closing driver instance 1308741443
17:51:58.920 [main] ERROR o.s.web.context.ContextLoader:313 - Context initialization failed

error from

datahub-gms

pod

is there a setting somewhere to disable telemetry on GMS? I'm in an intranet environment and the default telemetry enabled=True slows down the ingestion to rest-endpoint (unless i

datahub telemetry disable

) Also there is a daily entry in the logs about the system trying to reach api.mixpanel.com

Hello all I'm using this guide for Azure OIDC authentication https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react-azure - I have a problem with restarting the datahub frontend.

docker-compose -p datahub -f docker-compose.yml -f docker-compose.override.yml  up datahub-frontend-react
ERROR: Version in "./docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the `services` key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see <https://docs.docker.com/compose/compose-file/>

I have docker version 20.10.12 so it should match the requirements

Hello again I got the integration to Azure authentication running. Out of the box it seem that every user from an organization is accepted - what is the best way to restrict only access to some users ? Is there a configuration on datahub side necessary ? We have done this by a security group - so the question was answered already.

hi ,team How to switch login authentication to internal employee authentication。What is the way to replace datahub login with custom authentication。

hello, build datahub-gms, it appears : "#7 84.09 ERROR: Failed to create usr/lib/jvm/java-1.8-openjdk/jre/lib/rt.jar: Connection aborted #7 84.09 ERROR: openjdk8-jre-lib-8.302.08-r1: BAD signature"

Good morning, I am having some trouble with the configuration of the environment variables of the service datahub-actions. My problem is that, although I set the following variables in the docker-compose:

KAFKA_BOOTSTRAP_SERVER=broker1:9092 
KAFKA_SCHEMAREGISTRY_URL=<http://schema-registry:8081>      
KAFKA_PROPERTIES_SASL_KERBEROS_SERVICE_NAME=kafka
SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_PLAINTEXT
KAFKA_PROPERTIES_SASL_JAAS_CONFIG=com.sun.security.auth.module.Krb5LoginModule required principal='datahub/<realm>@<realm>' useKeyTab=true storeKey=true keyTab='/keytab/datahub.keytab'

and they are succesfully wrote as env variables (because when I execute the command

env

they appear), the actions service keeps printing the following error:

FAIL|rdkafka#consumer-1| [thrd:broker1:9092/bootstrap]: broker1:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 340ms in state UP, 3 identical error(s) suppressed)

Why is this error printed?? It might be related to the fact that the variables are not read correctly, but I don't understand why it happens only on that specific variable. I am using v0.8.41 for both the gms and front, for the actions the release 0.0.4 for the

acryldata/datahub-actions

image and 0.8.41 version for the CLI.

Hello team, I’m looking to deploy a custom datahub-action with custom transformations and actions. Do I need to clone the datahub-actions repo to build a custom image or is there a simpler way?

Hello team, is there a way to run datahub on Minikube M1 chip ? Any advise or pointers would help.

Hi team I am enabling the dbt validation tab by adding the run results file in test results path The recipe is running successfully but i am unable to my assertion and test in validation tab

Hi All, newbie here. I am trying to install datahub on docker in a no internet machine. Docker data hub quickstart won't work because of network restrictions, I do have access to docker hub through portainer to pull images. How can I install the data hub stack offline in the machine, please?

Hello team, why is it that a data hub recommends on using Kubernetes with helm charts for production? can we not use docker images?

This message was deleted.

I changed all 8080 ports to 8082 in docker-file.yml due to port conflict. As a result, Datahub frontend can not connect Datahub gms.

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to datahub-gms:8082 [datahub-gms/172.19.0.12] failed: Connection refused (Connection refused)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)

at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)

at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)

at client.AuthServiceClient.generateSessionTokenForUser(AuthServiceClient.java:79)

... 48 common frames omitted

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at <http://java.net|java.net>.PlainSocketImpl.socketConnect(Native Method)

at <http://java.net|java.net>.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at <http://java.net|java.net>.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at <http://java.net|java.net>.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at <http://java.net|java.net>.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at <http://java.net|java.net>.Socket.connect(Socket.java:607)

at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)

... 58 common frames omitted

The Datahub-gms status is healthy

CONTAINER ID   IMAGE                                   COMMAND                  CREATED          STATUS                    PORTS                                        NAMES

5f935d4604e6   confluentinc/cp-schema-registry:7.2.0   "/etc/confluent/dock…"   20 minutes ago   Up 20 minutes             8081/tcp, 0.0.0.0:8083->8083/tcp             schema-registry

99469c610934   linkedin/datahub-frontend-react:head    "datahub-frontend/bi…"   20 minutes ago   Up 20 minutes (healthy)   0.0.0.0:9002->9002/tcp                       datahub-frontend-react

e4b171aa3170   acryldata/datahub-actions:head          "/bin/sh -c 'dockeri…"   20 minutes ago   Up 20 seconds                                                          datahub_datahub-actions_1

d9383eaa2be6   confluentinc/cp-kafka:7.2.0             "/etc/confluent/dock…"   20 minutes ago   Up 20 minutes             0.0.0.0:9092->9092/tcp                       broker

2cf00e0e9973   linkedin/datahub-gms:head               "/bin/sh -c /datahub…"   20 minutes ago   Up 15 minutes (healthy)   8080/tcp, 0.0.0.0:8082->8082/tcp             datahub-gms

4618ce4b6d3c   elasticsearch:7.9.3                     "/tini -- /usr/local…"   20 minutes ago   Up 20 minutes (healthy)   0.0.0.0:9200->9200/tcp, 9300/tcp             elasticsearch

722a0d2b0e42   confluentinc/cp-zookeeper:7.2.0         "/etc/confluent/dock…"   20 minutes ago   Up 20 minutes             2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zookeeper

7119d1633395   mariadb:10.5.8                          "docker-entrypoint.s…"   20 minutes ago   Up 20 minutes             0.0.0.0:3306->3306/tcp                       mysql

This message was deleted.

Hi, I have updated datahub (v0.8.24 -> 0.8.41) using

helm upgrade datahub datahub/datahub

Helm version changed, new pod has been created, but the version on web frontend didn’t changed. Where did I miss something? Please, help

Hi All, just had a quick question regarding the datahub-actions container. Is this container needed for UI based ingestion to function?

Hi everyone. We're working on deploying Datahub to AWS and we're running into something odd. The directions tell us to create two load balancers, but when we run the

helm install datahub from datahub/datahub

command, two "classic" load balancers are created for us. Is this expected? https://datahubproject.io/docs/deploy/aws/#setup-datahub-using-helm . One loadblancer is created for the frontend and one for the api. There is no ingress set up on the EKS cluster.

Hey All, I’m running a datahub POC on GKE with helm and was wondering how folks were doing authn/authz? My current plan is to set the frontend as a load balancer and then use the google oidc auth. Thoughts?

Hi all, Yesterday, I tried to deploy DataHub on minikube by following this guide https://github.com/acryldata/datahub-helm#quickstart. But I got this error when deploy DataHub.

INSTALLATION FAILED: failed pre-install: timed out waiting for the condition.

My minikube resource: 6CPU - 8GB Memory. Any suggestions would be greatly appreciated 🤗

^^ And if it is still required, does anyone know if any team has successfully found a workaround to avoid the confluent schema registry dependency?

How can I deploy my local change to Kubernetes in the cloud? Looking for some documentation and pointers. Thank you!

hi, it seems datahub-gms container is stopped due to the dependencies not coming up in time. I mean gms waits dependencies to become up and running. As I observed dependencies are starting in 4 mins, and datahub-gms default timeout-threshold seems 4 min 🙂 . Do you guys know where can I change the timeout-threshold for gms server?

Hey everyone, we've successfully deployed datahub on AWS using EKS and datahubs helm chart. We were curious about how you recommend managing the "default" user. Can we remove this user altogether? Is this handled in the user.props file?

Hi team We are upgrading the datahub to 0.8.41 version While upgrading acryldata/datahub-ingestion image there is no corresponding exact 0.8.41 tag Other 0.8.41 tags are there so which one we need to use ?

In datahub v0.8.40 frontend screen goes blank whenever I click on glossary terms. Can anyone help on this? Thanks!

I was trying to setup LDAP for datahub-frontend, I was able to configure using com.sun.security.auth.module.LdapLoginModule and also org.eclipse.jetty.server.server.plus.jaas.spi.LdapLoginModule but these modules doesn't have the option to get email, first name, and other user attributes, hence I cannot get the users list inside datahub-frontend so I'm not able to create groups. Can anyone please tell how to get these user attributes or is there any Java module that can do this, please share the sample configuration if possible, thanks!