Hello

Hello team!! I'm new to the Datahub, I wanted to ingest CSV and make these observability checkpoints with the help of the Datahub, Is this possible for this? • Freshness • Volume • Scheme

Hello team, i am integrating datahub with dbt and i have a question here: let me introduce myself, i am a data engineer who are currently trying to integrate dbt and datahub for documentation purposes. Right now, I am exploring about how to insert terms, tags, and owners at datahub by defining them through dbt script, i have tried using meta and meta mapping but it seems like the tags and terms were only assigned to the "properties" column, not inserted directly to the "owner" or "terms" bar, did I miss something..? Thanks in advance!

version: 2

models:
    - name: payment
      description: "Final Datamart for Payment"
      meta_mapping:
        team_owner :
          match: ".*"
          operation: "add_owner"
          config:
            owner_type: group
        business_owner :
          match: ".*"
          operation: "add_owner"
          config:
            owner_type: user
            owner_category: BUSINESS_OWNER
        data_steward :
          match: ".*"
          operation: "add_owner"
          config:
            owner_type: user
            owner_category: DATA_STEWARD
        terms_list:
          match: ".*"
          operation: "add_terms"
          config:
            separator: ","
      meta:
        data_steward: Ilham
        team_owner: Payment

Hey team, Not able ingest data via unity catalog Config:

source:
    type: unity-catalog
    config:
        workspace_url: ''
        stateful_ingestion:
            enabled: true
        token: ''

Error:

ConfigurationError: unity-catalog is disabled; try running: pip install 'acryl-datahub[unity-catalog]

Hi Team - I have question that , whenever there is change in table structure at sources or added any tag , glossary term to the metadata . Need to get notification for same on email or to API . I tried to use action framework but not able to log table structure changes .Please Suggest ,TIA !

What are the minimum required permissions for an oracle ingestion? We operate on the principle of least access and will be using a svc account with the absolute minimum permissions possible for oracle ingestion. Its not mentioned in the docs…

Hi Team I have created custom data platform as below but this is not visible in the landing page of the Datahub , do i need to do any other changes ? curl 'http://localhost:8080/entities?action=ingest' -X POST --data '{ "entity":{ "value":{ "com.linkedin.metadata.snapshot.DataPlatformSnapshot":{ "aspects":[ { "com.linkedin.dataplatform.DataPlatformInfo":{ "datasetNameDelimiter": "/", "name": "nds_platform", "type": "OTHERS", "logoUrl": "https://<your-logo-url>" } } ], "urn":"urnlidataPlatform:MyCustomDataPlatform" } } } }'

Howdy Hubbers, I think I've found a bug with stateful ingestion enabled on MySQL recipes (at least for version 8.45 and later). Seems

DatahubIngestionCheckpointClass

is returning both

schema_pattern

and

database_pattern

in its response and throwing an error when a Checkpoint class instance is attempted to be created from that. How do you want the rest of the details...file a Github Issue?

Hellow! I wonder why I can't see nifi's full lineage in datahub.

image.png,image.png

Hello Team, We are working on spark integration and stuck at few areas to use it, the following are details 1. With one of the spark command - SaveIntoDataSourceCommand (jdbc) -> observed the the same action details getting overriding in Task, unable to see multiple run metrics 2. Spark integration are working to post details to datahub. But few jobs have exception to connect to server with InsertIntoHadoopFsRelationCommand and SaveIntoDataSourceCommand (jdbc) commands

Dear support, I have an issue with Looker, LookML in the new 0.9.0 version. Looker sees all objects of my Looker, Lookml builds relations with DWH. But Looker and LookML dont link with each other and I see a view from both ends not linked. I used the standard forms to create ingestions. What should I do to link those Lookml with Looker ? Thanks for your help!

Hi team, may I ask if I can update inputDatajobs and fineGrainedLineages of DataJobInputOutput respectively?

Hello, team. Is it possible now to ingest custom source via UI ingestion?

Hello team, I have been working with the Assertions and in the guide I have found a code that allows the creation of an entity on a column of a DataSet, however I have some doubts about this type of entity: Question 1: Do assertions work as a quality engine that can be created for any column of a dataset and executed on demand from the datahub itself, giving a % of validity? Question 2: Can assertion executions be scheduled? If someone could help me with the doubts or give me some more information about Assertions, it would be of great help. Thank you very much.

hi y'all, I was wondering if there is a way when running the ingestion to change the name with a transformer, thx! 🙏

I have few question about datahub action framework. I have implemented the action framework . But need some clarification on following : 1. if we add new table to sources , will it reflected by action framework pipeline ? 2. if there is changes (adding column to source table ) in column /field , will it be reflected in action framework pipeline ? 3.need to confirm those , I have airflow dag jobs running for the ingestion everyday at 06:00 at that time only it will reflect only right ?

Please suggest

Hello all, can anyone help me out on how to ingest fine grained lineages to datahub?

I have upgraded datahub to 0.9.2 and tried running lineage_emitter_dataset_finegrained.py file but in UI I am unable to view column level lineages

Dear support, please comment this message "You will need to run

lookml

ingestion through the CLI after you have ingested Looker metadata through the UI. Otherwise you will not be able to see Looker Views and their lineage to your warehouse tables." in https://datahubproject.io/docs/generated/ingestion/sources/looker/#config-details. Do I right understand that it means that LookML doesnt work with UI ? If it is so then is there any way or plans to make it working ? Thank you very much in advance!

Hello all, I was wondering if we have the option of specifying the path of the datasets in the sink that we are ingesting, for example:

source paths: <s3://my/bucket/dataset/*>
sink paths: Datasets/env/my/custom/path/* (instead of Datasets/env/s3/my/bucket/dataset/*)

Thanks!

Would it be beneficial to add the possibility to ingest tags from Snowflake tables/columns? would be happy to contribute, if it’s seen as a good idea

Hello Datahub Team!! Currently trying with the Postgres Ingestion, After ingestion, getting ingested and counts as well, But Datasource not reflecting in the UI. Any possible causes ???

Hello Team I want to add description information for schema fields in batches, how to do it? Rest client or other way? Are there any relevant examples for reference? thanks.

Are there plans to enable ingestion for GCS similar to support for S3 https://datahubproject.io/docs/generated/ingestion/sources/s3 ?

Hi team, Trying to integrate the lookml with datahub, it is writing nearly 48 entries but it is not showing on the datahub UI, Can anyone please suggest me about my error

This is my recipe.dhub.yml file

sink:
    type: datahub-rest
    config:
          server: '<https://datahub>-------.com:8080'
          token : XXXXXX

source:
    type: lookml
    config:
        github_info:
             repo: 'XXXXXX'
             # deploy_key_file: /home/----/---/id_ed25519        <-- This is not working, giving error "Extra Parameter"

        api:
            base_url: "<https://----looker.----.com:19999>"
            client_secret: 'XX'
            client_id: 'XX'
        base_folder: /home/--/----