Hi all, trying to find the manual way to ingest demoData ? could you confirm which account to use in datahub docker ingest-sample-data ? TIA

Hello Team, I noticed the ingest command is calling this endpoint: https://track.datahubproject.io/mp/engage could you explain what it does? or what is it tracking?

Oops I will delete because of cross-posting

datahub login with azure ad, is it possible? 🤨 I was checking this page, but i'm not sure if i'm right https://datahubproject.io/docs/generated/ingestion/sources/azure-ad/

Hi Everyone!

is there a way we can display transformations on the field level lineage ?

Hello Team, i’m getting error when building datahub-ingestion image

com.commercehub.gradle.plugin:gradle-avro-plugin:0.8.1

it seems like there are no more support for this plugin in maven repositories that you wrote at

repositories.gradle

Am i only person going through this issue?? I’ll appreciate for any guide. Thanks Below is error log what i get

A problem occurred configuring root project 'datahub-src'.
#8 88.91 > Could not resolve all artifacts for configuration ':classpath'.
#8 88.91    > Could not find com.commercehub.gradle.plugin:gradle-avro-plugin:0.8.1.
#8 88.91      Searched in the following locations:
#8 88.91        - <https://plugins.gradle.org/m2/com/commercehub/gradle/plugin/gradle-avro-plugin/0.8.1/gradle-avro-plugin-0.8.1.pom>
#8 88.91        - file:/root/.m2/repository/com/commercehub/gradle/plugin/gradle-avro-plugin/0.8.1/gradle-avro-plugin-0.8.1.pom
#8 88.91        - <https://repo.maven.apache.org/maven2/com/commercehub/gradle/plugin/gradle-avro-plugin/0.8.1/gradle-avro-plugin-0.8.1.pom>
#8 88.91        - <https://packages.confluent.io/maven/com/commercehub/gradle/plugin/gradle-avro-plugin/0.8.1/gradle-avro-plugin-0.8.1.pom>
#8 88.91        - <https://linkedin.jfrog.io/artifactory/open-source/com/commercehub/gradle/plugin/gradle-avro-plugin/0.8.1/gradle-avro-plugin-0.8.1.pom>
#8 88.91      Required by:
#8 88.91          project :
#8 88.91

Hi everyone, I want to use openapi to send request to datahub. I've generated an access token on webui (see the image below). And the token has been added in the request url (see the image below). But it still got 401 unauthorization (see the image below). Could any masters help me~ plz~~

Hello Team, I have a query like how can I make changes to datahub frontend code for the deployment I had made in the EC2 instance in AWS. I want to add few tabs in the frontend design for our custom use case.

Hello. Can I use LDAP based authentication of the UI after running docker quickstart? I made my own custom jaas.conf file and added the path of it to the volume of the container "datahub-frontend-react" in docker-compose.yml located in the quickstart directory. After doing this, I did docker-compose up datahub-frontend-react container and check login of my company's ldap/ad account but failed to login. What am I doing wrong?

I am running "datahub docker quickstart", but It stops while pulling docker images... (I waited about 10 minutes.) What's the problem?

Hi all! How can I generate token during CI / CD process or install it? As I see now my first token I have to generate in UI. If I use graphql mutation I already authorized

Hello Team, We are working on spark integration and stuck at few areas to use it, the following are details 1. With one of the spark command - SaveIntoDataSourceCommand (jdbc) -> observed the the same action details getting overriding in Task, unable to see multiple run metrics 2. Spark integration are working to post details to datahub. But few jobs have exception to connect to server with InsertIntoHadoopFsRelationCommand and SaveIntoDataSourceCommand (jdbc) commands

Hi 😊! Is there any way to create mlModel with the OpenAPI? (If there is a way, can you give me an example?) Thanks!!!

Hello Team, Just a question about column level lineage on datahub, I'm using the java emitter to publish the lineage. Currently, I constructed the lineage by creating urns on column level and setting them up inside

FineGrainedLineageArray

and then I passed it inside this block

UpstreamLineage upstreamLineage = new UpstreamLineage()
                .setUpstreams(upstreamArray)
                .setFineGrainedLineages(fineGrainedLineageArray);

and then eventually add it in the aspect when submitting the

MetadataChangeProposalWrapper.builder()
  .entityType("dataset")
                .entityUrn(targetTblDatasetUrn)
                .upsert()
                .aspect(upstreamLineage)
                .aspectName("upstreamLineage")
                .build()

however, the relationship doesn't seem to reflect from the UI? I'm using 0.9.2 datahub. Any idea what am I missing? Thank you!

Hey all, is there a way in DataHub to set ‘Personas’ or access controls as a way to limit the results returned when searching? For example as a Data Analyst if I search for ‘Store Locations’ i only want tables and maybe dashboards to be returned, rather than pipelines, dbt models, ML models, apis, everything under the sun 😛 . I find so many catalog tools are overwhelming for people who arent SUPER data savy, due to the sheer volume of search results returned

Hello Team, when i use datahub-ingestion image v0.8.40( i need to use this because our deployment of datahub is v0.8.40 and later releases makes error when ingestion) ingestion image doesn’t recognize tableau module and says

acryl-datahub[tableau] is required

but i already installed that image in the building processs is there any clear way to resolve this? i tried to debug this situtation for few hours but it only took time … Any Guide would help

Hello all, is OwnershipType field going to become deprecated? if so when? is there a chance this does not get deprecated? This feature is useful to our organization because we specify business_owner and technical_owner to some of our tables.

Hi, Can anyone tell me what is MAE and MCE?

This message was deleted.

Hello Team, Could you please let me know is it possible to create a new tab in the dataset entity page and then display some custom metric data in the new tab dynamically from the snowflake table. I had created the new tab but how I can dynamically get the data from snowflake table to the new tab. Are there any api connectors in datahub to do so?

general question, my team is going through a migration period moving the ELT to use delta and dbt. Has anyone ever used a data cataloging tool to help them in this process? Does anyone have any advice on whether its a good idea to create a data catalog in parallel with the migration? thanks for any info!

FineGrainedLineage in DataJobInputOutput aspect. Hi team we are playing with FineGrainedLineage recently and successfully ingested it into DataJobInputOutput aspect. But in the UI portal we are not seeing field level lineage as expected. We also tried to directly wire upstream datasets with downstream dataset, and field level lineage can be displayed correctly in this case. Just wondering if this different behaviour expected? Further more we are also wondering what's the "ultimate look" if an airflow job has multiple data job upstreams while it also has multiple datasets. Based on our experiments it shows both, which is kind of redundant. Is there a way to allow user to choose only show related datasets or airflow jobs.

Hello Team, is it possible to extend OwnershipTypeClass with addional types?

hey team — head of data(science) at synctera. Joined in a pavlovian response to this. Keen to see what the community’s up to. We are PG -> Airflow+Airbyte->BQ->DBT-> Looker, Metabase, Google Sheets, React -> Humans

Hello Guys, can we track lineage of a job triggered on AWS EMR and EKS from Airflow in DataHub.. like from airflow to spark end to end?

Hello Team, I wonder what is this and how it can be used with datahub? https://github.com/acryldata/meta-world/tree/master/CaKE

Hello Team, any docs or examples on how to customize datahub frontend e.g., add new information tabs to a dataset or domain programmatically?

Hi team. Is there a sample athena ingestion that I can reference? I followed documentation and setup and ingestion but 0 assets are ingested. I did not find any samples in https://demo.datahubproject.io/

we have data in s3 and use glue for transformation. will datahub show lineage in this case?