https://datahubproject.io logo
Join Slack
Powered by
# getting-started
  • b
    what actuallyu data catalog do in the datahub?
    ✅ 1
    b
    • 2
    • 1
  • g
    Hi. My system is as below. nginx - datahub OIDC(google) environment is being set up. first try. log-in succeed Log out second try. 404 error new try. remove PLAY_SESSION cookie and log-in try and succeed I don't know what is the problem. help me. please The nginx conf is as follows.
    Copy code
    server {
        listen          443 ssl;
        server_name     <http://datahub.my.com|datahub.my.com>;

        ssl_certificate         /nginx_ssl_my_com.crt;
        ssl_certificate_key     /_wildcard_my_com_SHA256WITHRSA.key;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers '';
        ssl_prefer_server_ciphers on;
        ssl_session_cache    shared:SSL:10m;
        ssl_session_timeout  10m;

        location / {

                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Server $host;
                proxy_set_header X-Forwarded-For $remote_addr;

                proxy_headers_hash_max_size 1024;
                proxy_headers_hash_bucket_size 128;

                proxy_pass <http://backend_datahub>;

                proxy_ssl_session_reuse on;

                proxy_set_header Host <http://datahub.my.com|datahub.my.com>;

                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "Upgrade";

                proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;

                client_max_body_size 10M;
        }

        error_page   404 500 502 503 504  /50x.html;

        location = /50x.html {
            root   html;
        }
}

upstream backend_datahub {
    server  localhost:9002;
    keepalive 5;
}
    b
    • 2
    • 1
  • r
    Hello Good afternoon, QQ: I'm try to run locally Datahub in docker on a M1 machine, ever time I ran it it appears this errors:
    Copy code
    Unable to run quickstart - the following issues were detected:
- datahub-gms is still starting
- elasticsearch-setup is still running
- elasticsearch is running but not healthy
    ✅ 1
    b
    • 2
    • 5
  • r
    Do you have any tips to run it properly? I already setup 15 GB of Memory, 5 CPUs and 2.5 Swap Memory but it won't work as expected 😢
  • h
    Hi everyone! We are considering using datahub for our project. We might use this guide to deploy it to GCP with kubernetes. What concern us is the cost per month. I used this calculator to estimate costs. • Is the estimated cost in my analysis correct? (never used kubernetes before, I might have exceeded the number of pods) • How much does it cost to run your DataHub service? • Are you using a different deployment? Thanks 😅
    a
    b
    • 3
    • 3
  • b
    Copy code
    Hello everyone,

was able to successfully integrate a SAP Hana database as a data source. However, the documentation does not state which rights the database user must have.

Can anyone tell me the roles/permissions or point me to where to find this in the documentation.

Thanks to the group!

Rainer
    👁️ 1
    a
    • 2
    • 1
  • w
    hey everyone, hope u are having a lovely weekend! qq; while it is super easy to set up Datahub locally with Docker, wonder how is everyone's experience to integrate & ingest metadata from Morden Data Stack? my org is using these toolings: • Snowflake for DWH • dbt for Data Transformation • Airflow for Job Orchestration • MariaDB as the OLTP for the old monolith • Kafka for direction we are moving towards afaik Datahub support all of the above, just wanna estimate how much effort for building the integration and maintaining them
    ✅ 1
    a
    • 2
    • 2
  • d
    after setting docker I was able to access datahub
    g
    a
    • 3
    • 10
  • d
    now it says failed to login
  • w
    hey everyone, when I start quickstart,occur a error that can't find datahubpolicyindex_v2 index. How to fix this question? Thanks
    thank you 1
    ✅ 1
    a
    s
    • 3
    • 3
  • f
    Hi Team As part of my project, we would like to store lineage in a file and call this file in other environments to populate lineage information. Is there a best way to store this info ? Currently I am storing Mcp object as pickle file but due to security I have to avoid pickle
    ✅ 1
    e
    m
    • 3
    • 6
  • s
    Hi, I am facing issue will starting datahub. It says- datahub-gms is running by not yet healthy. How to fix this issue?
    ✅ 1
    b
    a
    • 3
    • 3
  • r
    Hi, I'm in trouble starting datahub using k8s. When I deploy datahub, there is error with datahub-gms. This is error log.
    Copy code
    [minikube@localhost datahub]$ kubectl describe pods datahub-datahub-gms-6667df7bdc-564g8 
Name:             datahub-datahub-gms-6667df7bdc-564g8
Namespace:        default
Priority:         0
Service Account:  datahub-datahub-gms
Node:             minikube/192.168.49.2
Start Time:       Tue, 07 Feb 2023 17:10:35 +0900
Labels:           <http://app.kubernetes.io/instance=datahub|app.kubernetes.io/instance=datahub>
                  <http://app.kubernetes.io/name=datahub-gms|app.kubernetes.io/name=datahub-gms>
                  pod-template-hash=6667df7bdc
Annotations:      <none>
Status:           Running
IP:               172.17.0.11
IPs:
  IP:           172.17.0.11
Controlled By:  ReplicaSet/datahub-datahub-gms-6667df7bdc
Containers:
  datahub-gms:
    Container ID:   <docker://5f9d02decd7dc211faa911d56017de8f47ef3eb0da9713bfcb99989e998c90a>c
    Image:          linkedin/datahub-gms:head
    Image ID:       <docker-pullable://linkedin/datahub-gms@sha256:85cf456fe4756fddcb5fc03f45b083002e293cd2a38ce7feba7307ee5db3f365>
    Ports:          8080/TCP, 4318/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 07 Feb 2023 17:38:36 +0900
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Tue, 07 Feb 2023 17:33:36 +0900
      Finished:     Tue, 07 Feb 2023 17:38:36 +0900
    Ready:          False
    Restart Count:  5
    Liveness:       http-get http://:http/health delay=60s timeout=1s period=30s #success=1 #failure=8
    Readiness:      http-get http://:http/health delay=60s timeout=1s period=30s #success=1 #failure=8
    Environment:
      ENABLE_PROMETHEUS:                 true
      MCE_CONSUMER_ENABLED:              true
      MAE_CONSUMER_ENABLED:              true
      PE_CONSUMER_ENABLED:               true
      ENTITY_REGISTRY_CONFIG_PATH:       /datahub/datahub-gms/resources/entity-registry.yml
      DATAHUB_ANALYTICS_ENABLED:         true
      EBEAN_DATASOURCE_USERNAME:         root
      EBEAN_DATASOURCE_PASSWORD:         <set to the key 'mysql-root-password' in secret 'mysql-secrets'>  Optional: false
      EBEAN_DATASOURCE_HOST:             prerequisites-mysql:3306
      EBEAN_DATASOURCE_URL:              jdbc:<mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>
      EBEAN_DATASOURCE_DRIVER:           com.mysql.cj.jdbc.Driver
      KAFKA_BOOTSTRAP_SERVER:            prerequisites-kafka:9092
      KAFKA_SCHEMAREGISTRY_URL:          <http://prerequisites-cp-schema-registry:8081>
      ELASTICSEARCH_HOST:                elasticsearch-master
      ELASTICSEARCH_PORT:                9200
      SKIP_ELASTICSEARCH_CHECK:          false
      ELASTICSEARCH_USE_SSL:             false
      GRAPH_SERVICE_IMPL:                neo4j
      NEO4J_HOST:                        prerequisites-neo4j-community:7474
      NEO4J_URI:                         <bolt://prerequisites-neo4j-community>
      NEO4J_USERNAME:                    neo4j
      NEO4J_PASSWORD:                    <set to the key 'neo4j-password' in secret 'neo4j-secrets'>  Optional: false
      UI_INGESTION_ENABLED:              true
      SECRET_SERVICE_ENCRYPTION_KEY:     <set to the key 'encryption_key_secret' in secret 'datahub-encryption-secrets'>  Optional: false
      UI_INGESTION_DEFAULT_CLI_VERSION:  0.9.1
      SEARCH_SERVICE_ENABLE_CACHE:       false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-m6jc4 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-m6jc4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  29m                   default-scheduler  Successfully assigned default/datahub-datahub-gms-6667df7bdc-564g8 to minikube
  Normal   Pulling    29m                   kubelet            Pulling image "linkedin/datahub-gms:head"
  Normal   Pulled     26m                   kubelet            Successfully pulled image "linkedin/datahub-gms:head" in 2m59.664678032s
  Normal   Created    26m                   kubelet            Created container datahub-gms
  Normal   Started    26m                   kubelet            Started container datahub-gms
  Warning  Unhealthy  21m (x8 over 25m)     kubelet            Liveness probe failed: Get "<http://172.17.0.11:8080/health>": dial tcp 172.17.0.11:8080: connect: connection refused
  Normal   Killing    21m                   kubelet            Container datahub-gms failed liveness probe, will be restarted
  Warning  Unhealthy  4m17s (x53 over 25m)  kubelet            Readiness probe failed: Get "<http://172.17.0.11:8080/health>": dial tcp 172.17.0.11:8080: connect: connection refused
    I think k8s initialDelaySeconds setting is wrong. How I fix this issue? Is this solve to edit values.yaml file? Somethings strange is that I succed deploy datahub in last weeks using same setting. 😭
    s
    b
    +2
    • 5
    • 19
  • p
    How can I ingest the sample metadata into helm installation? I found only the docker way.
    ✅ 1
    a
    • 2
    • 2
  • m
    Hi, which aws eks infrastructure to support Datahub?
    ✅ 1
    a
    • 2
    • 1
  • s
    Hi, I am trying to create a lineage Mysql table->Sink Connector->Kafka message how do I show this in datahub? Dataset->Datajob->Dataset? I am confused between DataJob and DataProcess
    ✅ 1
    m
    • 2
    • 2
  • f
    Users and Groups ================== • Is the recommended setup for Users/Groups and Privileges to: 1. Setup datahub so that "datahub" admin user can login 2. Implement SSO so that authenticated Users can then login 3. Ingest Users/Groups (via datahub admin user) from a 3rd party authentication mechanism (i.e. Azure AD) 4. Apply datahub Roles/Policies to the ingested Groups and/or Users - these will then apply to Users logging in via SSO (since SSO Users will match based on ingested User fields) Thanks, Pete
    a
    a
    +2
    • 5
    • 7
  • p
    set the channel topic: For general discussion & questions about DataHub. For Metadata Day event, join #metadata-day-spring-21
  • r
    Hello. I'm installing datahub using minikube in docker for a simple test. (by referring to the quickstart with k8s) However, the following error occurs in datahub-GMS. (status is running but does not have a ready pod and has a connection problem)
    Copy code
    NAME                                                READY   STATUS      RESTARTS        AGE
datahub-acryl-datahub-actions-76f5459c6c-nkc8x      1/1     Running     1 (2m9s ago)    7m59s
datahub-datahub-frontend-d89c96686-chxcn            1/1     Running     0               7m59s
datahub-datahub-gms-77cb7f874d-fvshc                0/1     Running     1 (2m28s ago)   7m59s
datahub-datahub-upgrade-job-bmrr5                   0/1     Error       0               7m59s
datahub-datahub-upgrade-job-nmj54                   0/1     Error       0               2m2s
datahub-datahub-upgrade-job-rb6ll                   0/1     Error       0               4m31s
datahub-datahub-upgrade-job-xhv2n                   1/1     Running     0               50s
datahub-datahub-upgrade-job-xkvm2                   0/1     Error       0               3m19s
datahub-elasticsearch-setup-job-fqjnz               0/1     Completed   0               9m59s
datahub-kafka-setup-job-mxbg6                       0/1     Completed   0               9m47s
datahub-mysql-setup-job-wthlk                       0/1     Completed   0               8m10s
elasticsearch-master-0                              1/1     Running     0               12m
prerequisites-cp-schema-registry-5f89dd4974-cvxfn   2/2     Running     0               12m
prerequisites-kafka-0                               1/1     Running     0               12m
prerequisites-mysql-0                               1/1     Running     0               12m
prerequisites-neo4j-community-0                     1/1     Running     0               12m
prerequisites-zookeeper-0                           1/1     Running     0               12m

[minikube@localhost datahub]$ kubectl describe pods datahub-datahub-gms-77cb7f874d-fvshc 
Name:             datahub-datahub-gms-77cb7f874d-fvshc
Namespace:        default
Priority:         0
Service Account:  datahub-datahub-gms
Node:             minikube/192.168.49.2
Start Time:       Wed, 08 Feb 2023 21:22:54 +0900
Labels:           <http://app.kubernetes.io/instance=datahub|app.kubernetes.io/instance=datahub>
                  <http://app.kubernetes.io/name=datahub-gms|app.kubernetes.io/name=datahub-gms>
                  pod-template-hash=77cb7f874d
Annotations:      <none>
Status:           Running
IP:               172.17.0.9
IPs:
  IP:           172.17.0.9
Controlled By:  ReplicaSet/datahub-datahub-gms-77cb7f874d
Containers:
  datahub-gms:
    Container ID:   <docker://99645772cacad19af5d3c102a221e7cbb1748baa9769706165cd5296dc44a01>a
    Image:          linkedin/datahub-gms:head
    Image ID:       <docker-pullable://linkedin/datahub-gms@sha256:d2c8a7fd6075f9efa53cbd7a3bd9a58a6de1f242101db84ca7aa7d71b3f8d17e>
    Ports:          8080/TCP, 4318/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Wed, 08 Feb 2023 21:33:55 +0900
    Last State:     Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Wed, 08 Feb 2023 21:28:25 +0900
      Finished:     Wed, 08 Feb 2023 21:33:55 +0900
    Ready:          False
    Restart Count:  2
    Limits:
      memory:  2Gi
    Requests:
      cpu:      100m
      memory:   1Gi
    Liveness:   http-get http://:http/health delay=180s timeout=1s period=30s #success=1 #failure=5
    Readiness:  http-get http://:http/health delay=180s timeout=1s period=30s #success=1 #failure=5
    Environment:
      ENABLE_PROMETHEUS:                 true
      MCE_CONSUMER_ENABLED:              true
      MAE_CONSUMER_ENABLED:              true
      PE_CONSUMER_ENABLED:               true
      ENTITY_REGISTRY_CONFIG_PATH:       /datahub/datahub-gms/resources/entity-registry.yml
      DATAHUB_ANALYTICS_ENABLED:         true
      EBEAN_DATASOURCE_USERNAME:         root
      EBEAN_DATASOURCE_PASSWORD:         <set to the key 'mysql-root-password' in secret 'mysql-secrets'>  Optional: false
      EBEAN_DATASOURCE_HOST:             prerequisites-mysql:3306
      EBEAN_DATASOURCE_URL:              jdbc:<mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>
      EBEAN_DATASOURCE_DRIVER:           com.mysql.cj.jdbc.Driver
      KAFKA_BOOTSTRAP_SERVER:            prerequisites-kafka:9092
      KAFKA_SCHEMAREGISTRY_URL:          <http://prerequisites-cp-schema-registry:8081>
      SCHEMA_REGISTRY_TYPE:              KAFKA
      ELASTICSEARCH_HOST:                elasticsearch-master
      ELASTICSEARCH_PORT:                9200
      SKIP_ELASTICSEARCH_CHECK:          false
      ELASTICSEARCH_USE_SSL:             false
      GRAPH_SERVICE_IMPL:                elasticsearch
      UI_INGESTION_ENABLED:              true
      SECRET_SERVICE_ENCRYPTION_KEY:     <set to the key 'encryption_key_secret' in secret 'datahub-encryption-secrets'>  Optional: false
      UI_INGESTION_DEFAULT_CLI_VERSION:  0.9.6
      SEARCH_SERVICE_ENABLE_CACHE:       false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dddgd (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-dddgd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  11m                    default-scheduler  Successfully assigned default/datahub-datahub-gms-77cb7f874d-fvshc to minikube
  Normal   Pulling    11m                    kubelet            Pulling image "linkedin/datahub-gms:head"
  Normal   Pulled     11m                    kubelet            Successfully pulled image "linkedin/datahub-gms:head" in 13.012530772s
  Normal   Killing    6m17s                  kubelet            Container datahub-gms failed liveness probe, will be restarted
  Normal   Created    6m16s (x2 over 11m)    kubelet            Created container datahub-gms
  Normal   Started    6m16s (x2 over 11m)    kubelet            Started container datahub-gms
  Normal   Pulled     6m16s                  kubelet            Container image "linkedin/datahub-gms:head" already present on machine
  Warning  Unhealthy  2m17s (x7 over 8m17s)  kubelet            Liveness probe failed: Get "<http://172.17.0.9:8080/health>": dial tcp 172.17.0.9:8080: connect: connection refused
  Warning  Unhealthy  77s (x13 over 8m17s)   kubelet            Readiness probe failed: Get "<http://172.17.0.9:8080/health>": dial tcp 172.17.0.9:8080: connect: connection refused
    First time, I thought this was a problem with initialDelaySeconds. So with the help of other users, I created a value file with initialDelaySeconds and tried to install it. (180s) 
    helm install datahub datahub/datahub --values ./datahub/datahub-values.yaml
    However, the same error continues to occur. The same error occurs even if I change my network environment. What I don't understand is that last week I succeeded in installing and collecting with exactly the same settings as now. Is there anyone who can give me a solution? For your information, I am using CentOS7 in VM. I attach the error log and the used file. Thank you in advance.
    datahub-values.yaml
    ✅ 1
    a
    b
    +2
    • 5
    • 10
  • w
    Hi! I would like to manage the provisioning of some users and some policies from code (not interacting with the UI). • For the policies, I could add them here https://github.com/datahub-project/datahub/blob/master/metadata-service/war/src/main/resources/boot/policies.json Is there any other location to add additional policies? • For the users, is there anything similar? Why this? The use case I'm trying to solve is: I want to grant some service credentials with permissions to update some datasets and I want to follow infra as code approach and no UI if possible. So in the lack of service credentials, I was thinking on adding the users and the policies in those bootstrap files (even at the cost of re-build and deploy). WDYT? any better alternative to this?
    a
    m
    e
    • 4
    • 13
  • a
    Reposting here for visibility: https://datahubspace.slack.com/archives/C02FKQAGRG9/p1675878808830989 CC: @brainy-tent-14503
  • r
    Hello! I am just getting started with datahub and have it deployed via Kubernetes. I have synced it with a Kafka datasource. I see all the relevant kafka topics, but when drilling into them, I am not seeing any relevant schema information for those topics. I do have my schema registry integrated as well, so I had assumed the schemas would be synced. Is there anything I need to do to get my topic’s Avro schemas ingested properly? Thanks!
    ✅ 1
    m
    s
    • 3
    • 12
  • b
    I'm using "*" to do a entity search in the demo site, but am only able to return 3 tag results after filtering by tags? 🤨 There should be far more listings being returned. Or is * no longer a useable wildcard search term
    a
    • 2
    • 3
  • s
    Hello All, I am trying to use Datahub to capture lineage for a simple data processing within snowflake from raw table to transform and then to final table using task and streams. Can you please let me know why i am not seeing the lineage for the tables established ?
    a
    n
    • 3
    • 3
  • c
    Hello! We're very new to datahub. We've set up datahub quickstart in a python venv inside local windows machine, and want to try file-based ingestion. What recommended directory to put .yml file so that it can be seen? afaik docker containers can't just arbitrarily access local filesystem folders
    ✅ 1
    m
    • 2
    • 3
  • w
    Hi All, I am new to DataHub and Python. It is deployed in GCP Kubernetes. Want to extract the metadata from a json file. Plus extract the Lineage, map between the source and sink etc., Dont know where to start and end. If any direction on this will help. Thanks!
    ✅ 1
    a
    • 2
    • 3
  • w
    Hello, I´m trying to deploy Datahub using docker in ec2(aws) using DataHub quickstart. After install Docker and run the comand ( datahub docker quickstart ). I´m receiving: "Docker doesn't seem to be running. Did you start it?"
    ✅ 1
    b
    • 2
    • 5
  • c
    Hello!!! What would be the best way to have two instance of Datahub (one productive and one for Dev). Is there a way to transfer a Recipe via CI CD?
    a
    • 2
    • 2
  • c
    Hi Team, Question 1: About SSO, do we support PingFederate? If support, could you help to provide a config example? We are trying to deploy Datahub on our Kubernetes cluster, by https://github.com/acryldata/datahub-helm and it seems that only 
    google/okta/azure
    are supported.
    Copy code
    # OIDC auth based on <https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react>
oidcAuthentication:
  enabled: false
  # provider: google/okta/azure  <- choose only one
    https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-frontend/values.yaml On official doc, there is no PingFederate Provider-Specific guide either. https://datahubproject.io/docs/authentication/guides/sso/configure-oidc-react Question 2: Can I remove sso login part on UI, if my service has not configured SSO?
    ✅ 2
    👀 1
    i
    • 2
    • 4
  • w
    Hi, the datahub quickstart starts without using neo4j, so i have few questions regarding this: For what purpose does datahub use neo4j? , If i would like to have neo4j for my datahub setup what is the process ?
    ✅ 1
    m
    • 2
    • 2
1...545556...80Latest