I'm getting:

datahub-gms                | ERROR: No such classes directory file:///etc/datahub/plugins/auth/resources

error and datahub-gms is not starting up on quickstart. Anyone got any clue?

Hi everyone! I'm trying to debug a

redshift_usage

ingestion, but I'm getting hardly any logs. I tried both python sdk in airflow and cli. In airflow I only see this log, both in successful and failed ingestions:

INFO - Setting gms config

When I run the cli I also see this:

INFO     {datahub.cli.ingest_cli:91} - Starting metadata ingestion

The problem I'm facing is that the process ends because it consumes all the available ram, so I'd like to find which tables are the ones that cause these memory requirements. Is it possible that this callback isn't working as expected?

Good day. I have managed to setup datahub on GKE using the default setup guide. I am trying to setup bigquery metadata ingestion from the UI. I keep running into this error "{'lineage-gcp-logs': ['Error was 403 POST https://logging.googleapis.com/v2/entries:list?prettyPrint=false: Permission '\n" ' "\'logging.views.access\' denied on resource (or it may not exist). [{\'@type\': "\n' ' "\'type.googleapis.com/google.rpc.ErrorInfo\', \'reason\': \'IAM_PERMISSION_DENIED\', \'domain\': ' '\'iam.googleapis.com\', "\n' ' "\'metadata\': {\'permission\': \'logging.views.access\'}}]"]}" The service account I use has logging.views.access permission as well. Can someone help me troubleshoot this?

Hallo, i want the coworker to access the Datahub UI through a normal url and not use the IP:9002. where can i add our Domain / URL ? We use a Docker Compose environment.

Hi guys, I got gradlew build failed at

Class com.datahub.authorization.ranger.RangerAuthorizerTest

. Any suggestions to bypass or fix this stuff? Thanks

java.lang.UnsatisfiedLinkError: /Users/146685/Library/Caches/JNA/temp/jna4049579015533420670.tmp: dlopen(/Users/146685/Library/Caches/JNA/temp/jna4049579015533420670.tmp, 0x0001): tried: '/Users/146685/Library/Caches/JNA/temp/jna4049579015533420670.tmp' (fat file, but missing compatible architecture (have 'i386,x86_64', need 'arm64e')), '/usr/lib/jna4049579015533420670.tmp' (no such file)

Note: I built locally on Mac M1 machine.

Hi everyone, I’m trying to create a new entity type called

Metric

in DataHub following the fork approach. The Metric entity only has one key aspect called MetricKey. However,

./gradlew build

is failing in the

:metadata-io:test

step. The test report indicates four failures, all with the same error:

java.lang.IllegalArgumentException: Aspect MetricKey does not exist

Any idea to debug this? more info on 🧵

Hi everyone. I see performance problem trying to expand the nodes of the lineage graph. We have more than 300 datasource, each datasource has schema with 20-3000 fields. There are upstream and downstream job nodes linked with datasources, All job nodes are part of the same dataflow. When I expand the node I see the GraphQL API responses more then 20MB - 30MB each. Is this a problem with graphQL fragments design or incorrect modeling (all jobs in one flow, or loops in the graph) ? Datahub version 0.8.40

Hello everyone

A newb here trying to start datahub but running into this error

Unable to run quickstart: - Docker doesn't seem to be running. Did you start it?

but docker seems to be running fine

any pointers here

image.png

Hello everyone, I have been trying to migrate our Proof of Concept demo in Docker, to EKS with managed Kafka, Elastic Search and MySQL database. Upon making an attempt, I keep running into an error in

datahub-gms

which fails the pod creations with the following error I'll add as a reply. Has anyone run into this before, or have any ideas? If there's any information that I can provide, please let me know. Thanks in advance for any information you can provide.

Hi Team I am trying to enable the postgres in datahub. I am facing one error that postgres is trying to connect to mysql. Do we need to add new script in .values and .charts file? Please resolve it on priority. Please explain me how to enable postgres in datahub

Hello. When I try to install airflow lineage plugin to my airflow instance (airflow 2.3.4 deployed using docker-compose. added the library as part of the additional requirements param), I get the following error. Could anyone please help me with this?

Issues was posted in getting started - https://datahubspace.slack.com/archives/CV2KB471C/p1661359292314979

Hey everyone, So in an incremental attempt to solve a problem I had previously posed earlier, I was trying to just deploy the base datahub version through k8s helm on AWS. I also tried on minikube and encountered the following error when following the instructions on the website:

install.go:178: [debug] Original chart version: ""
install.go:195: [debug] CHART PATH: /home/sguillory/.cache/helm/repository/datahub-0.2.91.tgz

client.go:310: [debug] Starting delete for "datahub-elasticsearch-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job datahub-elasticsearch-setup-job with timeout of 5m0s
client.go:568: [debug] Add/Modify event for datahub-elasticsearch-setup-job: ADDED
client.go:607: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition
helm.go:84: [debug] failed pre-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
        <http://helm.sh/helm/v3/cmd/helm/install.go:127|helm.sh/helm/v3/cmd/helm/install.go:127>
<http://github.com/spf13/cobra.(*Command).execute|github.com/spf13/cobra.(*Command).execute>
        <http://github.com/spf13/cobra@v1.4.0/command.go:856|github.com/spf13/cobra@v1.4.0/command.go:856>
<http://github.com/spf13/cobra.(*Command).ExecuteC|github.com/spf13/cobra.(*Command).ExecuteC>
        <http://github.com/spf13/cobra@v1.4.0/command.go:974|github.com/spf13/cobra@v1.4.0/command.go:974>
<http://github.com/spf13/cobra.(*Command).Execute|github.com/spf13/cobra.(*Command).Execute>
        <http://github.com/spf13/cobra@v1.4.0/command.go:902|github.com/spf13/cobra@v1.4.0/command.go:902>
main.main
        <http://helm.sh/helm/v3/cmd/helm/helm.go:83|helm.sh/helm/v3/cmd/helm/helm.go:83>
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581

Can someone tell me what I am doing wrong?

Hi Team I want to understand how to enable validation feature and queries feature in datahubon DataHub.

Hi everyone! I want to know if I can download the new version of the Grafana dashboard? Looks like the dashboard from the GitHub repo has old metrics that aren’t showing up and I can’t find a replacement for some of them.

Hi folks, i tried to update datahub using helm

helm upgrade --install datahub datahub/datahub -f helm_custom_settings.yaml --version 0.2.90

but i got this error

Error: failed pre-install: timed out waiting for the condition

Hi- Is it possible to add glossary term to a dataset array or nested object in datahub using the UI?

Hey everyone, I recently created a fresh install of Datahub on Kubernetes. Everything started up fine and my team and I were able to ingest a bunch of Greenplum datasets, LDAP users & groups, and a number of glossary terms. However, now I'm trying to ingest an oracle database and running into a weird issue. The ingestion job completes successfully, but the ingested datasets fail to show up in the UI. I've checked in the MySQL database and the data is present there, so the entities are being created. I also ran the Elasticsearch re-indexing job which completed without error, but that did not produce any changes. We tried querying Elasticsearch itself to check for oracle data, but haven't been able to get it to play nice with the filters so we haven't been able to verify it's presence there. Has anyone run into something similar or have ideas on things we can do to diagnose and correct this issue?

Please advise. *kube:*client.go299 [debug] Starting delete for "sg-rcube-datahub-elasticsearch-setup-job" Job *kube:*client.go128 [debug] creating 1 resource(s) *kube:*client.go528 [debug] Watching for changes to Job sg-rcube-datahub-elasticsearch-setup-job with timeout of 1h23m20s *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: ADDED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 3, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 4, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 5, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*client.go595 [debug] sg-rcube-datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 6, jobs succeeded: 0 *kube:*client.go556 [debug] Add/Modify event for sg-rcube-datahub-elasticsearch-setup-job: MODIFIED *kube:*Error: failed pre-install: job failed: BackoffLimitExceeded *kube:*helm.go88 [debug] failed pre-install: job failed: BackoffLimitExceeded

Hello, When i make a the datahub docker quickstart i get the following error. Somebody know which Problem it could be?

Hello, Running the

datahub docker quickstart

I get the following error. (CentOS Linux 7 (Core) Kernel: Linux 3.10.0-957.el7.x86_64)

zookeeper is up-to-date
mysql is up-to-date
Starting elasticsearch ... 
broker is up-to-date
Starting mysql-setup   ... 
datahub-gms is up-to-date
schema-registry is up-to-date
datahub_datahub-actions_1 is up-to-date
datahub-frontend-react is up-to-date
Starting elasticsearch ... error
WARNING: Host is already in use by another container

ERROR: for elasticsearch  Cannot start service elasticsearch: driver failed programming external connectivity on endpoint elasticsearch (30d2cd08b59735e4a2125ca934d92258aa187167918d4f135e29c776fc62978b): Error starting userland proxy: listen tcp4 0.0.0.0:9200: bind:
Starting mysql-setup   ... done

ERROR: for elasticsearch  Cannot start service elasticsearch: driver failed programming external connectivity on endpoint elasticsearch (30d2cd08b59735e4a2125ca934d92258aa187167918d4f135e29c776fc62978b): Error starting userland proxy: listen tcp4 0.0.0.0:9200: bind: address already in use
ERROR: Encountered errors while bringing up the project.
.............
Unable to run quickstart - the following issues were detected:
- elasticsearch-setup container is not present
- datahub-gms is running but not healthy
- elasticsearch is not running

No other docker images are installed

Hi all, I am trying to have local Dev environment on my Mac M1 chip using minikube. I tried to bring up the prerequisites with helm installation. Please find the screenshot below for the error.

Hi, so I tried installing on a different server and got this:

# python3 -m datahub docker quickstart
/usr/local/lib/python3.6/site-packages/datahub/__init__.py:23: FutureWarning: DataHub will require Python 3.7 or newer in a future release. Please upgrade your Python version to continue using DataHub.
  FutureWarning,
[2022-08-29 13:44:30,077] ERROR    {datahub.entrypoints:189} - Command failed with 'DockerClient' object has no attribute 'close' In Docker SDK for Python 2.0, this method is now on the object APIClient. See the low-level API section of the documentation for more details.. Run with --debug to get full trace
[2022-08-29 13:44:30,077] INFO     {datahub.entrypoints:192} - DataHub CLI version: 0.8.43 at /usr/local/lib/python3.6/site-packages/datahub/__init__.py

python3 is 3.6.8

hey how i can make a lineage between mysql TABLE and redshift TABLE ??

how i can delete a Lineage ??