Hi guys, I've seen that you have in your Q2 roadmap to improve the way new entities and relationships are added. Any idea when it's coming? Better put, do I need to spend some effort to do it now, or could I wait few weeks? Thanks a lot!

Is it possible to use other DBs other than MySql for internal use of DataHub? I already have a msql server in prodution, it will be maybe usful to use it withiut spawning a new DB

Hi team, does the dataset have the version support? I know the MySQL stores the different version of aspects?

<!here> Reminder this is happening in < 12 hrs!

overlyaing KV store on top of a relational store may lead to performance issues. I don’t know if this is true of course..

Thanks to all who attended the Townhall! Here are the slides from the NoCode talk: https://docs.google.com/presentation/d/1_F2xy2agb7M4IFhAKy_ORte0w-7hBRfTCypxwWhr2kg/edit?usp=sharing

Hi All, https://datahubproject.io/docs/quickstart -> I am following this document to install datahub with docker, but I am hitting docker-compose version issues on AWS linux, whats the recommended version of docker compose? Also, the step 3 says, clone “this” repo, without a hyperlink, that needs to be fixed as well…thank you for any help…

<!here>

Just a suggestion, could we tag docker image IDs to datahub releases instead of relying on ${DATAHUB_VERSION:-latest}?

Is there a way to export data that was manually mapped/adjusted in datahub if it's necessary to tear-down a datahub instance and rebuild? IE if someone manually updated column descriptions or added tags and we needed to export this data somehow to re-load (assuming we need to teardown/delete volume and rebuild datahub)

@loud-island-88694 @mammoth-bear-12532 it was good to meet with you both yesterday, i found it very informative among several things, you mentioned you have a Datahub SaaS offering. couple of questions: • would we be able to add our own entities to it? (via code or no-code) • would we be able to extend the system? (eg if we wanted to make a change to the search functionality)

Is there a link that describes how to install the datahub CLI tool mentioned here: https://datahubproject.io/docs/debugging/#how-can-i-confirm-if-all-docker-containers-are-running-as-expected-after-a-quickstart?

Good afternoon friends. I am currently researching the data catalog and metadata tools for my company. I can see that a lot of subscription-based (paid) software offer AI-driven data discovery techniques and other "fancy" and advanced features. Our goal is, however, way more simple: We need a tool that would let us give meaningful descriptions to the databases and tables, assign owners for the specific DBs/tables, have an intuitively looking list of our databases with basic metrics such as • min value • max value • null count • outliers • other data quality metrics we may come up with for each table. Would that be possible to achieve with DataHub? How much do you think it would take in terms of time to deploy and configure the app? The reason I am asking such a basic question is the fact that I've already tried the demo and it doesn't seem to show every single thing I have on my requirement list above. Also, I've tried Amundsen, but that's such a fresh product - there is no way it's even close to satisfying at least a part of what we need.

This has probably been discussed before, but I just ran into this issue again and thought I’d ask about it. the issue is that, at least to my understanding, that the GMS doesn’t trigger a MAE if the incoming MCE doesn’t bring about an update in the aspects in the GMS database. We had some intermittent kafka problems (updated the cluster) and some aspects got registered in the GMS, but the MAE failed due to connection problems. I had do manually go in and delete the relevant rows in the database and then re-ingestion the data in order to get the MAE to go through. Is it currently possible to tell the GMS to retrigger the MAEs? If not, would it be possible to write a short script to do that?

Hey, Is it possible to deploy Datahub without Kafka, using only the push Api model?

Hello Can we use oracle19c as the backend for datahub.

Hey folks, I'm very interested to see what the integration with DQ systems such as Great Expectations look like. Are you still expecting to be able to release it before the end of June?

I am trying to run the MAE on K8s and it fails when it tries to check if there is a dataplatformindex_v2 on elasticSearch. I am using basic auth to connect to ES. I can see the initial connection uses the basic auth and returns a 200 but then appears to not use it when trying to verify the index. Is there a setting I should check?

Do we expect RBAC to be introduced soon? my colleagues are pondering if we should look at hacking the codebase or we should hold off and wait a while more.

Any reference step by step guide to install data hub on AWS

Happy Friday! I am curious how others are using the tag feature for tables and fields. My company has yet rolled out Datahub, but through previous experience of using tags (eg Jira) we thought that without proper governance the tagging feature could get messy very soon. 🙏 cc @silly-dusk-92062

Hello everyone. Not sure if it's just me, but I attempted to download the repo and get this error -

λ git clone <https://github.com/linkedin/datahub.git>

Cloning into 'datahub'...

remote: Enumerating objects: 85868, done.

remote: Counting objects: 100% (8205/8205), done.

remote: Compressing objects: 100% (1696/1696), done.

remote: Total 85868 (delta 3797), reused 7725 (delta 3577), pack-reused 77663

Receiving objects: 100% (85868/85868), 77.88 MiB | 3.23 MiB/s, done.

Resolving deltas: 100% (44752/44752), done.

error: invalid path 'metadata-models/src/main/pegasus/com/linkedin/metadata/aspect/ServiceAspect.pdl '

fatal: unable to checkout working tree

warning: Clone succeeded, but checkout failed.

You can inspect what was checked out with 'git status'

and retry with 'git restore --source=HEAD :/'

Is anyone else getting the same error? Thanks!

Hello, for our use-case I would have to add two fabrics (test, preprod) to datahub, to have consistent browsing paths/naming. I have found that one has to add these fabrics to an enum in various places and rebuild, or has the new noCode metadata update changed something in this regard?

would like to check if my understanding is correct, for the docker/dev.sh script, it will pull docker images with the #debug tag for containers specified in docker-compose.dev.yml? I got confused because I went to Docker hub and was unable to find any images when i searched by debug tag. Just did a docker-compose pull and it returned errors for all the debug images

Hello, able to share the roadmap for datahub beyond June 2021?

Hi, looks like the quickstart guide has just changed to use the datahub CLI as opposed to the quickstart.sh script? However, if I try to use the datahub CLI I get 'no such command docker'. datahub --version returns acryl-datahub 0.8.1.1 datahub --help shows 'check' 'ingest' and 'version' but indeed no 'docker' command....

Hello , I was setting up the datahub with these commands , and after setting up I am getting these following errors .

python -m pip install --upgrade pip wheel setuptools
python -m pip uninstall datahub acryl-datahub || true  # sanity check - ok if it fails
python -m pip install --upgrade acryl-datahub

When I check the version datahub --version it is sending this error . Anything I am missing here ?

RuntimeError: no validator found for <class 'datahub.ingestion.source.kafka_connect.KafkaConnectLineage'>, see `arbitrary_types_allowed` in Config

<!here> curious to learn what went into building the Lineage feature in DataHub? Want to see where DataHub's Lineage is headed in the future? Check out the blog post I wrote about DataHub’s Lineage Explorer. Give it a read and share it around to spread the DataHub love! https://medium.com/datahub-project/data-in-context-lineage-explorer-in-datahub-a53a9a476dc4

@green-football-43791 During Metadata ingestion we have observed duplicate column values for same urn and aspect ( except for version and createdOn date )for many of the records in aspect table. While analyzing the reason behind we got to know the source of ingestion was responsible for such entries. Can we have some kind of configurable check on ldh side to check for such duplicity before ingesting the data to mysql.

Hello! Could you tell me please how to debag avro.schema.AvroException: ('Datum union type not in schema: %s', 'com.linkedin.pegasus2avro.dataset.DownstreamLineage')