Hello everyone, I tried datahub using the quickstart command and now I want to deploy it in a server using docker compose, but it seems that there isn't one docker-compose that corresponds to the quickstart deployment, but it looks like I can generate it with this python file generate_docker_quickstart.py, am I right?

Quick one, how would I follow this if I've previously used the public helm chart?

Hello, some other questions: • I assume its not, but just wanting to confirm: is it possible to run datahub without kafka or ES? What prompts this question is that I'm trying to figure out the most lightweight implementation possible so far! • Is the EBEAN_DATASOURCE_PASSWORD in datahub-gms the password of the user datahub on the database backend? • are the head tagged images safe to use in prod or I am better going with the version (0.8.16 now)? Ty for the patience & help!

Hi everyone! Wondering if anyone has replaced

ElasticSearch

with

Elastic Cloud

on GCP. Doing a PoC for my company and trying to see how easy/hard it would be to swap out the dependencies.

as jaas doc says , i need to modify user.props file of datahub-frontend directory?

IMG_20211109_172824.jpg

i run dev.sh and got this error

I have a question regarding Policies: When creating policies, one can define in the second screen ("Configure Privilieges") to which resources the policy should be applied to (field "Resoures"). How can I apply the policy e.g. to all tables of a certain DB? When trying to add resources I can add only individual tables. Is there any support of wildcards or another way to apply a policy to all datasets/tables within a schema/DB (without adding all tables one by one to the policy)?

hey all, not specific to datahub, but wondering what the best way is to set the nodeSelector value when deploying with helm

Hi all! When deploying the whole stack using the Helm chart, aren’t we supposed to deploy the MCE and MAE jobs? I can’t find them running, am I missing something?

Hi team, we are exploring how to rebuild images for some custom code changes. Can I check what’s the suggested approach to rebuild below 4 images?

linkedin/datahub-gms
linkedin/datahub-frontend-react
linkedin/datahub-mae-consumer
linkedin/datahub-mce-consumer

Do we run

docker build

command with the corresponding Dockerfile?

Hi, currently we are trying to deploy datahub on AWS but our company policies deny creation of EKS with OIDC enabled, so we want to temporarily create one that does not uses cluster with OIDC- do you think it is possible with for example changing a little helm chart?

Hi team, i’ve passed custom topic names as extraEnvs in Helm values to

datahub-gms

, but still got below error

13:54:26.103 [main] ERROR o.s.web.context.ContextLoader - Context initialization failed
org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is java.lang.IllegalStateException: Topic(s) [MetadataChangeEvent_v4] is/are not present and missingTopicsFatal is true
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:185)
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:53)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:360)
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:158)

Do you have some idea?

Hello, I've got another question. Where should I change to add SSL certs and TLS vers? Could not find this in docs and I'm not sure how should I do it with java apps (don't know if its standarized). Could someone help?

Has anyone had any luck with GMS connecting to SR via TLS? I've tried to follow these docs https://datahubproject.io/docs/how/kafka-config#ssl and have also read https://github.com/linkedin/datahub/issues/1861 but haven't had any luck.. I am using the

head

container for GMS from Docker hub. My ENVs are as follows on the container:

KAFKASTORE_SSL_TRUSTORE_LOCATION: foo
KAFKASTORE_SSL_TRUSTSORE_PASSWORD: bar
KAFKASTORE_SECURITY_PROTOCOL: SSL
KAFKASTORE_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""

I've also tried using a few variations of these envs (

SCHEMA_REGISTRY, KAFKA

prefixes). Any insight would be appreciated! Thanks

Hello! Wondering is there any documentation on how to set up Datahub's Elasticsearch engine to consider Russian morphology in searches? We've been trying to use https://github.com/fooger/elasticsearch-analysis-morphology/, but unfortunately it doesn't work anymore.

Just a (simple?) question: What's the recommodation to protect the GUI via https/SSL? Just put a proxy in front of the datahub-frontend-react container and terminate the HTTPS there or are there other options/recommodations?

Hi everyone! How we can enable jmx-exporter on datahub components

Hi! I had Datahub v0.8.14 installed with Helm Chart v0.2.28. I updated to version 0.8.16 with Helm Chart v0.2.29 and after updating the last two rows of cards of the Datahub interface do not appear (Pipelines, Glossary of terms, Tables of characteristics, ML Models, ML Groups). I attach images with the difference. How can I see the Piplines section now? Thansk.

Hi Guys! Im trying out this tool, I have my resources on AWS.

While trying to ingest data from Athena, it doesnt find any table, I have double checked all the information and they are correct, can someonie help me with this?

Hi all - I've got a question on AWS deployments. I am following the guide here https://datahubproject.io/docs/deploy/aws . We're good on almost all the steps but are having some challenges on the schema registry side. We're trying to use Glue. The question is, can the catalog be in a different account? as in the config options I can only see

region

and

name

- unless the name is the Glue schema registry ARN?

Hey Team...I am getting

401

Error in

datahub-gms

in trying to connect to

OpenSearch

thru docker Following are the env variables i am using...

ELASTICSEARCH_HOST=<http://xxxxxx.es.amazonaws.com|xxxxxx.es.amazonaws.com>
ELASTICSEARCH_PORT=443
ELASTICSEARCH_USERNAME=username
ELASTICSEARCH_PASSWORD=password
USE_AWS_ELASTICSEARCH=true
ELASTICSEARCH_USE_SSL=true

When debugging thru curl the call is successful when

curl -u '<username:password>' '<https://xxxxxx.es.amazonaws.com:443>'

Where as from datahub-gms its trying to call as

'<https://username:password@xxxxxx.es.amazonaws.com:443>'

Have anyone faced this before?

A general question has anyone in the community have used the AWS Open Search as the index store successfully?

Quick Question - Can we fetch the stats for glue data source ?

Hello ! I am trying to deploying the Helm chart (Elastic only) but disabling the persistence of elasticsearch (

elasticsearch.persistence.enabled: false

) , kafka(

cp-helm-charts.cp-kafka.persistence.enabled: false

), zookeeper(

cp-helm-charts.cp-zookeeper.persistence.enabled: false

) and mysql (

mysql.primary.persistence.enabled: false

and

mysql.secondary.persistence.enabled: false

) to simplify my deployment (we don’t support persistence on our k8s server yet ), but it still deploys statefulset:

NAME                                          READY   AGE
statefulset.apps/elasticsearch-master         0/2     12m
statefulset.apps/prerequisites-cp-kafka       0/3     12m
statefulset.apps/prerequisites-cp-zookeeper   0/1     12m
statefulset.apps/prerequisites-mysql          0/1     12m

Any idea how to totally disable persistence and statefulness ?

Quick Question - I am using metadata ingestion tool to ingest the metadata for glue data source. There is one property env in the recipe file . What could be the possible values we can place there for environment s.

hey team, we tried to rebuild the images for frontend, gms, mae-consumer, and mce-consumer from the latest tag

v0.8.17

but encountered below error for

frontend

and

gms

pods when deploy over Helm.

mae-consumer

and

mce-consumer

pods run fine. Do you have some idea what goes wrong?

Error: dl failure on line 896
Error: failed /usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server/libjvm.so, because Error loading shared library libgcc_s.so.1: No such file or directory (needed by /usr/lib/libstdc++.so.6)

Hi, It’s the helm repo down? Maybe due to the latest realese? Maybe it’s my k8 instalation but..

"datahub" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "datahub" chart repository
Update Complete. ⎈Happy Helming!⎈
Error: failed to download "datahub/datahub-prerequisites" (hint: running `helm repo update` may help)

https://helm.datahubproject.io/ --> Navigation from the browser get a 404, but maybe that is normal

Hi , On v0.8.17 tag i am trying to run docker build on frontend module. But its breaking with below error .

Step 5/26: FROM --platform=$BUILDPLATFORM node:16.13.0alpline3.14 AS Prod-build
failed to parse platform : "" is invalid component of "":

Please suggest what is this $BUILDPLATFORM ARG is and should we add it in the Dockerfile ?

Quick question - How do we redirect to the right Airflow link from here ? Any configuration that needs to be done