Hi, Our company is very interested in integrating DataHub in our system. Is there anything required ? What is the sequentially steps in deploying DataHub ? How can we backup metadata ? If we can deploy by docker, what is the persistent part to backup metadata when docker restarts ? Can you share the experience in deploying ? We need a doccument how to do that and we really need your support. It would be nice , if we could contact in detail. Here is my email address: phamminhsyhcmus@gmail.com

<!here> 📣 DataHub now supports discovering highly popular tables and requires less resources to run! Join our June Townhall this Friday to hear more 🙂 • When: June 25th at 9am US PT 🕘 • Signup to get a calendar invite: here • Townhall Zoom: https://zoom.datahubproject.io Agenda: Project Updates by Shirshanka - Release notes - RBAC update - Roadmap for H2 2021 Demo: Table Popularity powered by Query Activity by @gray-shoe-75895 Case Study: Business Glossary in production at Saxo Bank by @cuddly-spoon-1635 @silly-london-3535 Developer Session: Simplified Deployment for DataHub by @big-carpet-38439 @green-football-43791

Hey guys, I came to DataHub after trying to install Amundsen for a month. I hope it would be easy to install and configure this than what I endured in the last couple of months.

<!here> We just published the May edition for our community newsletter yesterday. Thanks for all the contributions and the great momentum. Please share with your network and clap away 🙂 https://medium.com/datahub-project/linkedin-datahub-project-updates-ed98cdf913c1?source=friends_link&amp;sk=9930ec5579299b155ea87c747683d1ad

Hi all! Just wanted to say hi 👋, introduce myself and ask a few questions! Matthias from Arabesque AI here, we are looking into metadata stores and landed on Datahub and it's been very enjoyable so far 🎉. A few initial questions (please point me to documentation where possible, my initial searches might have been failing / too superficial to find it myself, but I am happy to go ahead and dig deeper myself before burdening more people 🙂) : • Is there a way to ingest metadata from Google Cloud Storage or ElasticSearch? • I read somewhere about the possibility to add ML models as well - is this documented somewhere on how to do so? • Lastly: data lineage - I've found docs on how to do so using Airflow, but is there a way to add this manually (for now) and how? We set it up internally on GCP, so I'd be happy to look into contributing docs/steps if that'd be useful!

Hi folks….I am looking to extend the GMS Client to get Datasets by specific version in addition to the Urn i.e.

Dataset get(Dataset urn, int version)

(https://github.com/linkedin/datahub/blob/master/gms/client/src/main/java/com/linkedin/dataset/client/Datasets.java#L62) Is this achievable? Could you guys highlight the high level changes that might be needed to provide this ability….

I got this error when i tried to install elasticsearch by kubernetes . I setup my server with configuration of 8 GB RAM. Could it be out of memory. How can i fix it ? I am using your configuration on minikube

Thanks for the cool presentations today! Folks if you have questions about the protobuf schema integration with business glossary, ping @silly-london-3535 and @cuddly-spoon-1635

I'll publish the roadmap for H2 2021 soon! please ping me for additions / collab opportunities 🙏

Hi Datahub team, what's the status of [no code modeling](https://datahubproject.io/docs/advanced/no-code-modeling). I looked at the code, the sample

service

entity is there. after I build & run, I tried the sample post request

url '<http://localhost:8080/entities?action=ingest>' -X POST -H 'X-RestLi-Protocol-Version:2.0.0' --data '{
   "entity":{ 
      "value":{
         "com.linkedin.metadata.snapshot.ServiceSnapshot":{
            "urn": "urn:li:service:mydemoservice",
            "aspects":[
               {
                  "com.linkedin.service.ServiceInfo":{
                     "description":"My demo service",
                     "owner": "urn:li:corpuser:user1"                     
                  }
               },
               {
                  "com.linkedin.common.BrowsePaths":{
                     "paths":[
                        "/my/custom/browse/path1",
                        "/my/custom/browse/path2"
                     ]
                  }
               }
            ]
         }
      }
   }
}'

It is purely illustrative (of course if you need this model you can add it)

Hey everyone, I just joined the townhall and I am extremely impressed by just about everything. I went straight ahead and tried to quickstart it but it seems to fail and I am not quite sure what the fix could be. I receive this error and I attached the log as requested. Any sort of help would be amazing.

Hi there, I was wondering if APIs could be considered as a Data Source so they could be included in the Lineage of the Data Sets? It could be possible to extract all the metadata from the

Swagger

documentation 🤔

Any way to ingest data from Tableau, Metabase and Redash and bring in the lineage? I would love to collaborate, but will need pointers to start with…

<!here> Release 0.8.4 (as seen in Friday's town-hall) is now available. (https://github.com/linkedin/datahub/releases/tag/v0.8.4) Release Highlights 🎉 • Dataset Popularity, Recent Queries powered by Usage logs (support for Snowflake, BigQuery) • Markdown descriptions and editing • New Integrations : Glue Jobs, Feast • Versioned API for metadata GETs • No neo4j requirement, Elastic for Graph • Docker image hardening • Improved logging • GCP Deployment Guide

curious: why are you interested in moving off of Neo4j?

<!here>: If you missed the Friday townhall, or just want to share it with your colleagues and friends, the full version of the townhall is now up!

I am doing

datahub docker quickstart

and it is pulling a bunch of things

Pulling elasticsearch          ... downloading (68.7%)
Pulling elasticsearch-setup    ... done
Pulling mysql                  ... pull complete
Pulling datahub-gms            ... pull complete
Pulling datahub-frontend-react ... done
Pulling mysql-setup            ... 
Pulling zookeeper              ... waiting
Pulling broker                 ... 
Pulling schema-registry        ... downloading (49.4%)
Pulling kafka-setup            ...

Are all of these hard dependencies of datahub?

How does datahub handle schema changes?

Hi all, I am getting those errors when trying to run DataHub v0.8.4 from quickstart.sh:

datahub-gms               | 2021/06/30 18:18:42 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: no such host. Sleeping 1s
mysql-setup               | 2021/06/30 18:18:43 Problem with dial: dial tcp: lookup mysql on 127.0.0.11:53: no such host. Sleeping 1s

We are trying to enable the OIDC (AWS cognito) authentication for dockerized Datahub We followed the instructions given in the below link https://datahubproject.io/docs/how/configure-oidc-react/ We have configured following properties from datahub/docker/datahub-frontend/env/docker.env Required OIDC configs AUTH_OIDC_ENABLED=true AUTH_OIDC_CLIENT_ID=XXXXXXXX AUTH_OIDC_CLIENT_SECRET= #AUTH_OIDC_DISCOVERY_URI=https://xxxxxxxxxxxxxxxxxxxxxxxx/.well-known/openid-configuration AUTH_OIDC_DISCOVERY_URI=https://xxxxxxxxxxxx.xxxxxxxxxxxxxxxx/openid-configuration AUTH_OIDC_BASE_URL=https://XXXXXXXXX.com Optional OIDC configs AUTH_OIDC_USER_NAME_CLAIM=email AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+) AUTH_OIDC_SCOPE=openid Uncomment to disable JAAS username / password authentication (enabled by defau lt) AUTH_JAAS_ENABLED=false But for some reasons redirection is not happening, instead it is taking us to default datahub login page when we hit the base url. Please let us know, if we are missing something Thanks in advance.

i see that the roadmap for Jul to Nov has been updated. would having the UI show historic version of schemas be part of the work to be expected as well?

Hi All. Could you point me to a correct direction? I want to write Java application, which could use DataHub classes (DAO, EntityClient, AspectClient) to work with DataHub data. So the questions about that: 1. Is there any example for such application? 2. How do we setup maven dependencies correctly (as only GMA is available in artifactory) for that app? 3. How do we setup required connections for that app (to primary SQL DB and secondary ES) in DataHub way correctly? Thanks!

I have an issue with the quickstart. I'm trying to start datahub on windows server 2016. I have docker fully running and docker-compose installed. Upon running "python -m datahub docker quickstart" I get several errors saying "ERROR: for kafka-setup image operating system "linux" cannot be used on this platform". Here is the full log of what I get upon running quickstart: https://pastebin.com/ffMVtVet. Trying to run with "docker-compose -p datahub up", I get the following error: "ERROR: no matching manifest for windows/amd64 10.0.14393 in the manifest list entries"

If I am reading this correctly https://datahubproject.io/docs/architecture/metadata-ingestion/ it is possible for me to remove the kafka dependency in ingestion if we decide to simply use the http push mechanism. We are not using kafka at the moment so don't want to introduce that currently, if possible.

Are the docs at https://datahubproject.io/docs/architecture/metadata-serving/ up-to-date? I read in recent changelogs that Neo4j is not longer required. I was wondering if kafka stream here is required or not.

This page https://datahubproject.io/docs/what/mxe/#metadata-audit-event-mae can be reached via linksc present in https://datahubproject.io/docs/architecture/metadata-serving/ but is not present on the sidebar when looking at docs page. Is this intentional?

Hello, We are evaluating Datahub and trying to estimate what is required to implement and maintain it. We are now evaluating how to extend the metadata model, since our reality is a bit different from what the model presents this is a big issue for us. The reason is we are mainly focused in small data and work with in-house developed tools. So my questions refer to the extension of the model. 1) Step number 6 of the documentation requires that the GraphQL & React models are extended to contemplate what was incorporated to the model in the previous steps. The question is, how is this an optional step? Are the new entities available for querying and the user interface if not done? 2) This is more of a conceptual question... Why is it necessary to redifine in GraphQL & React what was already defined in Pegasus? We see this as an error-prone task which could (should?) be automatic. Is there something we are missing? If not, is this already identified and is it on the roadmap? 3) Similarly, we don't understand why a rebuild in necesary after the model is extended (step 5). We are also concerned that it unnecesarily affects the production system to incorporate new concepts (we see it analogous to restarting a DMBS after a table create). Again, are we missing something here? Thanks a lot. Regards, Agustín Mullin

When compiling datahub frontend with 'gradlew build' I always seem to get a failed build with the error like this: https://pastebin.com/TUjU5shD. I have datahub running and have JDK8. Any suggestion would be very appreciated.

Hi We need to point to external kafka, mysql and elastic search from datahub, currently we point to dockerized versions of them. Please let us know what all changes we need to do in order to achieve this.