Does the

mysql-setup-job

container only run this sql?

-- create datahub database
CREATE DATABASE IF NOT EXISTS DATAHUB_DB_NAME;
USE DATAHUB_DB_NAME;

-- create metadata aspect table
create table if not exists metadata_aspect_v2 (
  urn                           varchar(500) not null,
  aspect                        varchar(200) not null,
  version                       bigint(20) not null,
  metadata                      longtext not null,
  systemmetadata                longtext,
  createdon                     datetime(6) not null,
  createdby                     varchar(255) not null,
  createdfor                    varchar(255),
  constraint pk_metadata_aspect_v2 primary key (urn,aspect,version)
);

-- create default records for datahub user if not exists
CREATE TABLE temp_metadata_aspect_v2 LIKE metadata_aspect_v2;
INSERT INTO temp_metadata_aspect_v2 (urn, aspect, version, metadata, createdon, createdby) VALUES(
  'urn:li:corpuser:datahub',
  'corpUserInfo',
  0,
  '{"displayName":"Data Hub","active":true,"fullName":"Data Hub","email":"<mailto:datahub@linkedin.com|datahub@linkedin.com>"}',
  now(),
  'urn:li:principal:datahub'
), (
  'urn:li:corpuser:datahub',
  'corpUserEditableInfo',
  0,
  '{"skills":[],"teams":[],"pictureLink":"<https://raw.githubusercontent.com/linkedin/datahub/master/datahub-web/packages/data-portal/public/assets/images/default_avatar.png>"}',
  now(),
  'urn:li:principal:datahub'
);
-- only add default records if metadata_aspect is empty
INSERT INTO metadata_aspect_v2
SELECT * FROM temp_metadata_aspect_v2
WHERE NOT EXISTS (SELECT * from metadata_aspect_v2);
DROP TABLE temp_metadata_aspect_v2;

-- create metadata index table
CREATE TABLE IF NOT EXISTS metadata_index (
 `id` BIGINT NOT NULL AUTO_INCREMENT,
 `urn` VARCHAR(200) NOT NULL,
 `aspect` VARCHAR(150) NOT NULL,
 `path` VARCHAR(150) NOT NULL,
 `longVal` BIGINT,
 `stringVal` VARCHAR(200),
 `doubleVal` DOUBLE,
 CONSTRAINT id_pk PRIMARY KEY (id),
 INDEX longIndex (`urn`,`aspect`,`path`,`longVal`),
 INDEX stringIndex (`urn`,`aspect`,`path`,`stringVal`),
 INDEX doubleIndex (`urn`,`aspect`,`path`,`doubleVal`)
);

Hi folks. Would you tell me if it’s possible to make a backup of the state of Datahub deployed on AWS EKS (without AWS manages services for the storage layer) and restore it in the case of an emergency? Or maybe using managed services for storage is a better option?

How can I apply OIDC on k8s datahub-frontend? Just add AUTH_OIDC env like applying OIDC to docker?

I've managed to get AWS Kafka/MSK working as a replacement for the k8s quickstart config and I was wondering if it was possible to use the AWS Glue Schema Registry instead of the prerequisites cp-schema-registry service? Can the schemaregistry config use an AWS ARN instead of a URL?

Hi community, we prefer postgres to mysql (aligns better with our stack) so I'd like to use postgres as backend for gms. However, the helm chart contains a lot of mysql-specific variables, and no mention of postgres. Does the helm chart support postgres? https://github.com/acryldata/datahub-helm/tree/master/charts/datahub

Is the version supposed to be unavailable?

docker run -it --rm --entrypoint=bash --tty linkedin/datahub-ingestion:v0.8.6  
datahub@2523f447d4db:/$ datahub version
DataHub CLI version: unavailable (installed editable via git)
Python version: 3.8.11 (default, Jun 29 2021, 19:54:56) 
[GCC 8.3.0]
datahub@2523f447d4db:/$

Is there a Docker image I can use for

./gradlew build

that has all necessary dependencies (Gradle, Python, etc.)?

What are the pros and cons of running the Mae and mce jobs standalone?

Hi community, I ran into an issue with the helm chart, where it defaults to appending

:9200

(the port) to the elastic URI, although the AWS ElasticSeararch doesn't work that way (it's reached just

<https://elastic-host.domain.com>

without the port). This causes the elasticearch init-job to fail,

2021/07/22 143855 Waiting for: https://myuser:mypassword@search-mydomain.eu-central-1.es.amazonaws.com:9200

I got past this step by setting it to yaml null, `~`:

global:
    elasticsearch:
        port: ~

However, gms does not heed the port variable, and continues to use

:9200

(same log as above). Are there some env vars I can set on GMS to make it not append a port?

Hello team, Can we integrate Microsoft Active Directory authentication to DataHub? is it possible?

I've checked out the new version of DataHub that's deployed on K8s and got error with mae component

@early-lamp-41924 Is this correct? I think it should be

with .Values.labels

https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-frontend/templates/deployment.yaml#L6 I am not 100% sure so wanted to ask. Similar includes are present in most of the files. And I don't think any of them are working. Neither

labels

nor

selectorLabels

are being applied to deployments

Hi Community! In DataHub v0.8.10, GMS throws this exception:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configEntityRegistry' defined in com.linkedin.gms.factory.entityregistry.ConfigEntityRegistryFactory: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: ../../metadata-models/src/main/resources/entity-registry.yml (No such file or directory)

I'm wondering if any of you have faced the same? Seems there is some yaml file that it is missing (or cannot find at that path).

Untitled

Hi Trying Dockerised Datahub in AWS env, as @blue-holiday-20644 suggested. Using Postgres instead of mysql. Getting this error - Trying the above with postgres rather than mysql- unfortunately getting this - Any suggestions?

I've been deploy DataHub on k8s and i want to track user activity over log of pods like gms or frontend. However the log of pod just give information over time but not date. And that's just not enough. I want to add date into log that will come along with time. Where i can edit DataHub code to add date into log INFO ?

Hi community. Am deploying to EKS and have just started seeing an error when applying prerequisite helm chart:

kubectl get pods -n datahub
NAME                                               READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                             0/1     Pending   0          16m
elasticsearch-master-1                             0/1     Pending   0          16m
elasticsearch-master-2                             0/1     Pending   0          16m
prerequisites-cp-schema-registry-cf79bfccf-zxp2d   1/2     Error     7          16m
prerequisites-kafka-0                              0/1     Pending   0          16m
prerequisites-zookeeper-0                          0/1     Pending   0          16m

Error coming from cp-schema-registry pod:

kubectl logs -n datahub -f pod/prerequisites-cp-schema-registry-cf79bfccf-zxp2d
error: a container name must be specified for pod prerequisites-cp-schema-registry-cf79bfccf-zxp2d, choose one of: [prometheus-jmx-exporter cp-schema-registry-server]

Has anyone seen this before?

General - I am deploying to EKS and getting the following error . Any guidance on this is greatly appreciated - helm install datahub datahub/datahub W0913 153351.098210  9497 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob W0913 153351.230126  9497 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob

im deploying the datahub containers as individual deployments in k8s....can there ever be more than 1 instance of the various containers, ie broker, ES, mySQL, GMS, frontend-react running at the same time?

Wondering if anyone has experienced this pain: Kubernetes clusters not working when a database fails over

Hey, I deployed Datahub 0.8.12 on our k8s cluster via the Helm chart 0.2.22, and it is working perfectly. Looked into monitoring via JMX Exporter, but couldnt figure out how to expose the metrics from datahub-gms in such a way that our existing Prometheus stack is able to pull them I noticed that in the latest version (0.8.12), monitoring stack via docker-compose is available, is there something missing in the Helm chart deployment?

Can someone pls look at - https://github.com/acryldata/datahub-helm/issues/24

General - Using this guide https://datahubproject.io/docs/deploy/aws#use-aws-managed-services-for-the-storage-layer I am able to setup the prerequisities up and running, but when I try to deploy datahub, only the following services are coming up.

Hi there! Any plan on releasing the new helm for v0.8.14?

General - Getting this when deploying Datahub as containers on EKS 0s     Warning  SyncLoadBalancerFailed  service/datahub-datahub-frontend         Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB 0s     Normal  EnsuringLoadBalancer   service/datahub-datahub-gms           Ensuring load balancer 0s     Warning  SyncLoadBalancerFailed  service/datahub-datahub-gms           Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB 0s     Warning  FailedBuildModel     ingress/datahub-datahub-frontend         Failed build model due to couldn't auto-discover subnets: unable to discover at least one subnet 0s     Normal  EnsuringLoadBalancer   service/datahub-datahub-frontend         Ensuring load balancer 0s     Warning  SyncLoadBalancerFailed  service/datahub-datahub-frontend         Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB 0s     Normal  EnsuringLoadBalancer   service/datahub-datahub-gms           Ensuring load balancer 0s     Warning  SyncLoadBalancerFailed  service/datahub-datahub-gms           Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB 0s     Warning  FailedBuildModel     ingress/datahub-datahub-frontend         Failed build model due to couldn't auto-discover subnets: unable to discover at least one subnet How do we resolve this ?

yes, use this datahub-datahub-gms:8080, works

Hey everyone We are stuck in quite a deadlock while trying to upgrade our datahub from v0.6 to v0.8.14. Here are the steps we have performed: 1. Took backup of our existing mysql db and launched a new mysql container and restored the dump into it 2. Using these helm charts: https://github.com/acryldata/datahub-helm to install the upgraded version. We install all the prerequisites (except Mysql because that we are using the one we launched in step 1) 3. All the prerequisites get installed properly. Then when we try to install datahub via helm chart, everything runs fine except

datahub-gms

and

datahubUpgrade

job.

datahub-gms

throws following error:

javax.persistence.PersistenceException: Query threw SQLException:Table 'datahub.metadata_aspect_v2' doesn't exist

and

datahubUpgrade

throws following error:

ERROR: Cannot connect to GMSat host test-datahub-datahub-gms port 8080. Make sure GMS is on the latest version and is running at that host before starting the migration.

Now both the errors seems to be dependent on each other to me. I was wondering if we are missing any step in between and does the

metadata_aspect_v2

table needs to be created manually?

Hi Team , I was doing some changes in UI look and feel of datahub. To verify those changes i was trying to build the docker image of datahub-frontend module on running the below command from datahub home folder

sudo docker build -t updated_datahub_frontend_react -f ./docker/datahub-frontend

But this end with the error below

---> Running in 55db64e7685d
/bin/sh: ./gradlew: not found
The command '/bin/sh -c cd datahub-src && ./gradlew :datahub-frontend:dist -PenableEmber=${ENABLE_EMBER} -x test -x yarnTest -x yarnLint     && cp datahub-frontend/build/distributions/datahub-frontend.zip ../datahub-frontend.zip     && cd .. && rm -rf datahub-src && unzip datahub-frontend.zip' returned a non-zero code: 127

Could anyone please help me resolve this issue.

And i guess error is because of yarn is not able to download the cypress package

Thank you so much once again for all your help and support. And sorry for such a silly issue, took you lots of time to dive into it.