https://datahubproject.io logo
Join Slack
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    Hi, actually I don't know anything about datahub yet. What documents should I start with? I ran Quickstart and added sample data.
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    Hi @calm-motorcycle-16283, welcome!
    Copy to Clipboard
    What kinds of metadata are you looking to integrate into Datahub?
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    Well, in fact, the data I am trying to integrate are data made of parquets. Is such data possible?
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    Is that stored as hive tables
    Copy to Clipboard
    Or just raw parquet files
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    For now, It is just raw parquet files. But I can move to hive tables.
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    Check out the metadata-ingestion scripts here. https://github.com/linkedin/datahub/tree/master/metadata-ingestion
    Copy to Clipboard
    There is a hive source
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    I have various datasources stored as parquets, and I join and clean them daily and save them as new parquets. Can this also be expressed in lineage? I want to manage these through datahub, but I am not sure how to do it.
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    What orchestration system are you using? Something like airflow?
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    Sorry for a very basic question.😅
    Copy to Clipboard
    Yes I use airflow.
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    Right now, we support coarse grain lineage where you have to emit an event connecting your inputs to your output tables.
    Copy to Clipboard
    If you can emit that, Datahub will store it and reflect it in the ui
    Copy to Clipboard
    There isn’t an airflow recipe for this yet, but it should be possible to create one.
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    If you don't mind, can you explain more deatil about?
    If you can emit that, Datahub will store it and reflect it in the ui.
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    So there are three pieces to this:1. The Event that you emit 2. DataHub stores the lineage in its backend 3. You can see the lineage in the UI
    Copy to Clipboard
    which of the three do you want to understand more about
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    1. The Event that you emit
    Copy to Clipboard
    specifically: L214
    Copy to Clipboard
    so the event that you emit would need to include the upstream-lineage(s) of the output dataset that you are creating
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    Oh good. Thanks.👍 Quick question, should someone make this json file manual? Or is it auto-generated from datasources or other stored data?
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    This json file is just an example of some sample metadata that you can quickly ingest into datahub
    Copy to Clipboard
    if you want to do it programmatically, you would probably create the event in memory and then emit it to datahub (either over Kafka or over http/REST)
    Copy to Clipboard
    if you want to figure out how to generate the event in Python, you should chat with @gray-shoe-75895 as he has been contributing changes in that area
    c

    calm-motorcycle-16283

    1 year ago
    Copy to Clipboard
    Okay thanks, I think I understand little by little now.
    m

    mammoth-bear-12532

    1 year ago
    Copy to Clipboard
    👍 we're here to help!
View count: 1
Join thread in Slack