Hi Team We use the DataHub Airflow lineage plugin for Airflow Datahub integration, we are trying to emit lineage using DatahubEmitterOperator (see the python script below), but it looks like it not working, we do not see any lineage info in DataJob mce event, inlet and outlet fields are empty Airflow ingestion log

[2023-02-21, 07:13:46 UTC] {_plugin.py:179} INFO - Emitted Start Datahub Dataprocess Instance: DataProcessInstance(id='redshift_lineage_emission_dag_emit_lineage_manual__2023-02-21T07:13:31.971338+00:00', urn=<datahub.utilities.urns.data_process_instance_urn.DataProcessInstanceUrn object at 0xffff9a989ac0>, orchestrator='***', cluster='prod', type='BATCH_AD_HOC', template_urn=<datahub.utilities.urns.data_job_urn.DataJobUrn object at 0xffff864bb790>, parent_instance=None, properties={'run_id': 'manual__2023-02-21T07:13:31.971338+00:00', 'duration': '0.375603', 'start_date': '2023-02-21 07:13:45.274840+00:00', 'end_date': '2023-02-21 07:13:45.650443+00:00', 'execution_date': '2023-02-21 07:13:31.971338+00:00', 'try_number': '1', 'hostname': '05badb3885d0', 'max_tries': '1', 'external_executor_id': 'None', 'pid': '3840758', 'state': 'success', 'operator': 'DatahubEmitterOperator', 'priority_weight': '1', 'unixname': 'default', 'log_url': '<http://localhost:8080/log?execution_date=2023-02-21T07%3A13%3A31.971338%2B00%3A00&task_id=emit_lineage&dag_id=redshift_lineage_emission_dag&map_index=-1>'}, url='<http://localhost:8080/log?execution_date=2023-02-21T07%3A13%3A31.971338%2B00%3A00&task_id=emit_lineage&dag_id=redshift_lineage_emission_dag&map_index=-1>', inlets=[], outlets=[], upstream_urns=[])

It works with BashOperator with setting inlets and outlets, we can see lineage in datahub lineage_emission_dag.py

"""Lineage Emission

This example demonstrates how to emit lineage to DataHub within an Airflow DAG.
"""

from datetime import timedelta

import datahub.emitter.mce_builder as builder
from airflow import DAG
#from airflow.providers.amazon.aws.operators.redshift import RedshiftSQLOperator
from contrib.operators.redshift_sql_operator import RedshiftSQLOperator
from airflow.utils.dates import days_ago

from datahub_provider.operators.datahub import DatahubEmitterOperator
global_vars = {
    'REDSHIFT_CONN_ID': 'redshift_prod'
}

default_args = {
    "owner": "de",
    "depends_on_past": False,
    "email": ["<mailto:de-team@zynga.com|de-team@zynga.com>"],
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
    "execution_timeout": timedelta(minutes=120),
}

with DAG(
        "lineage_emission_dag",
        default_args=default_args,
        description="An example DAG demonstrating lineage emission within an Airflow DAG.",
        schedule_interval=timedelta(days=1),
        start_date=days_ago(2),
        catchup=False,
) as dag:
    # This example shows a RedshiftSQLOperator followed by a lineage emission with DatahubEmitterOperator

    sql = """select count(*) as tables
            from information_schema.tables
            where table_type = 'BASE TABLE'"""
    redshift_transformation_task = RedshiftSQLOperator(
        task_id="redshift_transformation",
        dag=dag,
        redshift_conn_id=global_vars['REDSHIFT_CONN_ID'],
        sql=sql
    )
    # same DatahubEmitterOperator can be used to emit lineage in any context.
    datahub_emit_lineage_task = DatahubEmitterOperator(
        task_id="emit_lineage",
        datahub_conn_id="datahub_rest_default",
        mces=[
            builder.make_lineage_mce(
                upstream_urns=[
                    builder.make_dataset_urn("redshift", "warehouse.zbatch.tableA"),
                    builder.make_dataset_urn("redshift", "warehouse.zbatch.tableB"),
                ],
                downstream_urn=builder.make_dataset_urn(
                    "redshift", "warehouse.zbatch.tableC"
                ),
            )
        ]
    )

    redshift_transformation_task >> datahub_emit_lineage_task

It looks DatahubEmitterOperator is not working with latest release our setup Datahub version 0.9.6 acryl-datahub 0.9.6 acryl-datahub-airflow-plugin 0.9.6 Airflow v2.3.4