<@U01JUBAPZPS> During Metadata ingestion we have o...
# getting-startede
early-hydrogen-59749
06/21/2021, 12:20 PM@green-football-43791 During Metadata ingestion we have observed duplicate column values for same urn and aspect ( except for version and createdOn date )for many of the records in aspect table. While analyzing the reason behind we got to know the source of ingestion was responsible for such entries. Can we have some kind of configurable check on ldh side to check for such duplicity before ingesting the data to mysql.
g
green-football-43791
06/21/2021, 2:23 PMAre you saying the schema aspect has duplicate columns in the schema?
green-football-43791
06/21/2021, 2:23 PMI think for this case, it would be easiest to try and filter these out on the source side.
green-football-43791
06/21/2021, 2:24 PMAlthough I agree, we should have some constraints on DH side to verify that there are no two columns with an identical fieldPath
green-football-43791
06/21/2021, 2:24 PMDo you know why the source is emitting duplicate columns to begin with?
e
early-hydrogen-59749
06/22/2021, 5:40 AMIn the below screenshot UpstreamLineage aspect have duplicate entries for same dataset in which metadata information is same.
early-hydrogen-59749
06/22/2021, 5:43 AMWe follow push based architecture in which data is re-ingested and refreshed every 2 days.
early-hydrogen-59749
06/23/2021, 11:17 AM@green-football-43791 any feedback on this
g
green-football-43791
06/23/2021, 2:27 PMHey @early-hydrogen-59749 - we are looking into this. For now, I would recommend leaving the AuditStamp null in the SchemaMetadata. Separately, we will make some changes to prevent AuditStamp from being set in the future.