A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi, I am using the “newly” introduced way to ingest lineage data via REST with the *MetadataChangeProposalWrapper*.
I want to track the ingests of every lineage change via custom runIds but setting the systemMetadata seems to does not have any impact on this and I cannot track the changes (or either revert them in case of failure)
Is this a known issue or by design? Anybody else facing this or have an idea? Thanks :slightly_smiling_face:

<@U02AF5P6QDS>: this was fixed recently by -&gt; <https://github.com/linkedin/datahub/pull/3900>

I don't know if you are facing the same issue or this is a new / different one though

Oh thanks for the hint. I did a quick search but did not find that PR. I will try with the latest python package and see if this fixes it

<@UV0M2EB8Q> unfortunately its not the case. I tested it with `0.8.23.1`  and the following snippet but it does not get my provided runId neither some generated one if I send it without the systemMetadata. Do you have any other idea if I am doing something wrong or should I file an issue? Thanks :slightly_smiling_face:
```    lineage_mcp = MetadataChangeProposalWrapper(
        entityType="dataset",
        changeType=ChangeTypeClass.UPSERT,
        entityUrn=mce_builder.make_dataset_urn("bigquery", table_ref, settings.env),
        aspectName="upstreamLineage",
        aspect=upstream_lineage,
        systemMetadata=SystemMetadata(runId="my_lineage_ingestion_test")
    )```

I think an issue would be useful here - I’ll need to take a closer look 

<@U02AF5P6QDS>: are you sending over rest_emitter or kafka?

<@U01GCJKA8P9> <https://github.com/linkedin/datahub/issues/3978>

<@U02AF5P6QDS> I just tried to reproduce this

```
  chart_info_mcp = MetadataChangeProposalWrapper(
      entityType="chart",
      changeType=ChangeTypeClass.UPSERT,
      entityUrn=builder.make_chart_urn(platform="qlik", name="dummy"),
      aspectName="chartInfo",
      aspect=chart_info,
      systemMetadata=SystemMetadataClass(runId="my_lineage_ingestion_test")
  )```

```+--------------------------------------+--------+---------------------------+
| my_lineage_ingestion_test            |      1 | 1969-12-31 16:00:00 (PST) |
+--------------------------------------+--------+---------------------------+```

Note that if you've previously produced the aspect, the system metadata will only update if you change the aspect value!