early-hydrogen-27542
05/05/2023, 8:23 PMTier 1
term to a model via mapping, then swap it out for a Tier 3
term, both terms remain on the model. We have stateful ingestion enabled, and it works at the model level (e.g. soft deletes deleted models). Column meta mapping appears to be stateful.
2. How would I update my recipe to have column level terms prefaced with a term node (e.g. PII.
)? I would like to avoid the user having to type that prefix, but unsure how to add it when using add_terms
instead of just add_term
Version: 0.10.1
Recipe config:
meta_mapping:
datahub.owner:
match: ".*"
operation: "add_owner"
config:
owner_type: "group"
contains_pii:
match: True
operation: "add_term"
config:
term: "PersonalInformation.PII"
contains_pii:
match: False
operation: "add_term"
config:
term: "<http://PersonalInformation.No|PersonalInformation.No> PII"
tier:
match: "Tier 1|Tier 2|Tier 3"
operation: "add_term"
config:
term: "Tier.{{ $match }}"
column_meta_mapping:
glossary_terms:
match: ".*"
operation: "add_terms"
config:
separator: ","
lively-cat-88289
05/05/2023, 8:23 PMmodern-artist-55754
05/06/2023, 12:44 AMPatch
i.e. append meta. You can switch it to Override
mode which will replace existing one with the new one... Although i am not sure the implication with other metadata.
If you find restrictions in out of the box functionality, you can write your own custom transformer. You have a lot of flexibility in what to do with your metadatadelightful-ram-75848
05/08/2023, 2:43 AMearly-hydrogen-27542
05/08/2023, 12:17 PMmodern-artist-55754
05/08/2023, 12:37 PMadd_terms
.
https://github.com/datahub-project/datahub/blob/0e69e5a810d22d5f5eb076e261a23f893ec7e63d/metadata-ingestion/src/datahub/utilities/mapping.py#L238modern-artist-55754
05/08/2023, 12:39 PMearly-hydrogen-27542
05/08/2023, 12:50 PM