A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi, all! I'm setting up lineage collection from Airflow. Is it possible to pass some usage data (like number of records processed) along with the data flow?

How are you currently collecting information? Are you using the DataHubEmitterOperator <https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/example_dags/lineage_emission_dag.py#L54|as shown here>? Or lineage backend?

Hi, missed your message, sorry. Nope, I'm using inlets/outlets of standard Airflow operator, as it was shown in examples. This is a piece of code:

`from datahub_provider.entities import Dataset`

`UserLoadOperator(`
    `task_id='user-load',`
    `dag=dag,`
    `source_conn_id='jira_ts',`
    `destination_conn_id='ts_data',`
    `inlets={'datasets': [`
        `Dataset('jira.users', 'users'),` 
        `Dataset('jira.confluence', 'confluence')`
    `]},`
    `outlets={'datasets': [`
        `Dataset('postgres.timesheet2', 'public.user_data')`
    `]}`
`)`