Hello, can I get some eyes 👀 on this small PR for fixing mongodb ingestion when

platform_instance

is missing from recipe? Thank you!

Hi team, I am in the process of writing a custom ingestion source, using the datahub documentation as a guide. does the datahub server need to have the custom source installed? or can sources be shimmed in at the package level for whatever is communicating to the server?

Hi All, Request team to help with PR https://github.com/datahub-project/datahub/pull/9412 - This is basically addressing broken Spark lineage for hive tables. Message links on other channels posted for the same https://datahubspace.slack.com/archives/C04BSKCPXCK/p1702033468493729?thread_ts=1696582266.105799&cid=C04BSKCPXCK.

query = """ query searchAcrossLineage { searchAcrossLineage( input: { query: "*" urn: "urnlidataset:(urnlidataPlatform:""" + dataplatform + """,""" + dataset + """.""" + table + """,PROD)" start: 0 count: 10 direction: """ + lin_stream + """ orFilters: [{ and: [ { condition: EQUAL negated: false field: "degree" values: ["1","2", "3+"] } ] # Additional search filters can be included here as well } ] } ) { searchResults { degree entity { urn type } } } } """ How to add subTypes in this query, resutlting in subTypes like table or view

Hello Team, could someone point out where is the usage statistics extraction logic is in the source code for Teradata source ?: https://datahubproject.io/docs/features/dataset-usage-and-query-history/

Hello, can I get some eyes 👀 on this small pr https://github.com/acryldata/datahub/pull/274? It's about overriding datajob external_url in datahub airflow plugin. Thanks

Hi team, I'm using Datastream and BigQuery. The table on BigQuery is a upsert table. So, it will get a error when it execute query on https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/queries.py#L72 My question: Seem likes this query only collect the size of table, number of partition, etc. We can skip this metric. Do we have any suggestion to deal with it? We also can check exception and default a value in this case.

Hi team, since AWS has announced Amazon MSK IAM authentication now supports all programming languages, I want to check with the community on where we at for adopting this in datahub-action, the usage looks relatively straightforward as the example suggested here. Happy to put together a draft rfc if this hasn't been done yet.

Hi I have a very small fix for mongodb PR to fix when downsampling collection schema, the output is not always consistent. This is because the schema is sort by frequency, but there are chances frequency is the same and we should compare their

delimited_name

to make sure output is consistent.

Hi guys, any chance someone from the team could have a quick look at this open PR: https://github.com/datahub-project/datahub/pull/9212. We currently need 3 custom images (mae-consumer, datahub-upgrade and gms) because of it. Many thanks!

Hi guys! What to do, if some tests were failed due to network reason? And this tests are not related to main scope of changes of PR? For example. PR is related to fixing mssql data_flows & data_jobs, but failed tests are Airflow related? PR

my requirement creates a dataset for each database inside MySQL. Inside each database dataset, creates separate table datasets with names like "table_name" (without the database prefix), and each table dataset contains all the fields for that specific table. from the csv file useing python code is that possible i am getting databasename.tablename

Heya all, we (Checkout.com) built an access request feature through DataHub and was wondering if there’s any interest in receiving a contribution from us on it? This is what we’ve done: 1. Extend the metadata model to include an aspect called

accessRequest

and

accessRequests

where an

accessRequest

contains an

auditStamp

and a stringMap of

additionalMetadata

(similar to custom properties). 2. We attached

accessRequests

to

datasets

,

containers

and

dataProducts

3. We implemented a graphQL query

addAccessRequest

which takes looks like this

mutation addAccessRequest {
  addAccessRequest(input: {
    resourceUrn: "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)"
    additionalMetadata: [
      {
        key: "why2"
        value: "I need it"
      },
      {
        key: "who"
        value: "The Data Platform group"
      }
    ]
  })
}

4. We also implemented a button that contains a form that looks like (image attached) which submits the addAccessRequest ticket. (This could likely be made more modular by someone else) 5. Finally we have an action that runs that takes tickets and assigns them to a freshservice queue based on the domain attached (but we could contribute a simpler version that just requires a group_id and freshservice_api key in the config (or as secrets). Not sure which components of these would be useful for other people, but I thought it would at least start a discussion!

Hello everyone, I reimplemented the (S3) delta-lake ingestion source to support Azure (az://, adl://, abfs://, and will provide support for Microsoft Fabric as well), is there any interest for this code to be contributed back to the project once is ready?. Thanks!

We have EntityType.INGESTION_SOURCE in EntityType enum values but search operation over INGESTION_SOURCE return the exception:

"Unknown entity type: INGESTION_SOURCE"

. Note: Change the type to any other EntityType will return correct results

query SearchIngestionSourcesWithoutFilter {
  search(input: {
    type: INGESTION_SOURCE,
    query: "*", 
    start: 0,
    count: 10
    
  }) {
    start,
    count,
    total: count,
    
    searchResults {
      entity {
        urn
      }
    }
  }
}

Hello, I am struggling to understand why my PR fails during linting: https://github.com/datahub-project/datahub/actions/runs/7747136631/job/21126913171?pr=9762 Considering that: 1. I didn't touch mentioned files or anything related to them 2. Last time they were touched was long time ago 3. Running

:metadata-ingestion:lint

task succeeds locally Does anybody have any idea what is happening there? I notice master builds succeed...

Hey! How to trigger github actions? It seems getting stuck as always

Hello, here is the PR for upgrading

shiro-core

due to a vulnerability. please take a look. thanks

Hello, here is the PR for upgrading

fastapi

and

gitdb

due to a vulnerability. please take a look. thanks

datahubbbbHello, I have two improvement PR out for review: 1. implement flattening of struct fields in DynamoDB ingestion, this would allow flattening fields for

Map

attribute type in DynamoDB: https://github.com/datahub-project/datahub/pull/9852 2. improve sorting when downsampling collection schema, this would allow the output to be consistent with alphabetical order, relatively small: https://github.com/datahub-project/datahub/pull/9856 Would be greatly appreciate if team can have a look 🙏🙏🙏

Hi Team - can someone help in reviewing my PR https://github.com/acryldata/datahub-helm/pull/429 This PR would help in adding custom labels to datahub-frontend service.

Hi Team, Kindly review our PR for the introduction of new entity "Business Attribute" in Datahub

Hi team 👋 , could you please review my PR about fixing various lineage problems for the Tableau entities.

Hey folks 👋🏻! I found that some people other than me are interested in something like enforceable views - functionality for limiting entity visibility/discoverability. In company I'm working for we're still evaluating DataHub and also interested in having this feature - so we can dedicate some time and work on it. But before starting I just wanted to ask is there something similar in the works or on a roadmap? Thanks. https://datahubspace.slack.com/archives/CV2UXSE9L/p1680054010543709 https://datahubspace.slack.com/archives/CV2UXSE9L/p1702598510793429

Hi All. This is my first contribution on Datahub. Can anyone please review this PR and trigger workflow? https://github.com/datahub-project/datahub/pull/9921 This is about adding the support for JSONL files in s3 / GCS source.

Hi All @hundreds-photographer-13496 we need to update description for multiple tables across schema, database and schemas, we can use mutation api, but then i have to make a separate call for each record. is there a way /endpoint where we can supply urn and description in array and it will update description for multiple tables?

Hi. Is there some documented way to do local development for the Datahub Airflow plugin? I’m trying to spin up a local airflow via docker and mount the files for the airflow plugin into the airflow plugins directory. I’m generating the files after doing a

python setup.py build

on the datahub-airflow-plugin. Is this the right approach? I’ve also tried to install the necessary python dependencies but i’m getting errors from Airflow when it tries to load the plugin. I’m currently getting the following error:

File "/usr/local/airflow/plugins/datahub_airflow_plugin/datahub_listener.py", line 19, in <module>
aws-mwaa-local-runner-2_7-local-runner-1  |     from datahub.sql_parsing.sqlglot_lineage import SqlParsingResult
aws-mwaa-local-runner-2_7-local-runner-1  | ModuleNotFoundError: No module named 'datahub.sql_parsing'

This is despite me installing the

acryl-datahub[sql-parser]==0.12.1.5

package. Any guidance would be much appreciated!

Which python package would i need to install, to be able to import

datahub.sql_parsing

? I see that its part of the main acryl-datahub but for some reason i can’t do that import when i install 0.12.1.5

Could someone please review my PR below? It has been a week since I submitted. https://github.com/acryldata/datahub-helm/pull/431

We were thinking about adding a failed topic for the MAE consumer, but just had some questions before looking into that. Found that the following files for the MCE consumer have failed topics: https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mce-consumer/[…]/com/linkedin/metadata/kafka/MetadataChangeEventsProcessor.java https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mce-consumer/[…]m/linkedin/metadata/kafka/MetadataChangeProposalsProcessor.java However, this file for the MAE consumer doesn't send to a failed topic in the event of an error: https://github.com/datahub-project/datahub/blob/master/metadata-jobs/mae-consumer/[…]ava/com/linkedin/metadata/kafka/MetadataChangeLogProcessor.java Is there a reason that the MetadataChangeLogProcessor doesn't have a failed topic like the MCE consumer or is that something that we can look into?