:wave: folks - a couple questions around <dbt meta...
# troubleshoot
e
👋 folks - a couple questions around dbt meta mapping: 1. Is there a way to make model level meta mapping stateful? For instance, if I add a
Tier 1
term to a model via mapping, then swap it out for a
Tier 3
term, both terms remain on the model. We have stateful ingestion enabled, and it works at the model level (e.g. soft deletes deleted models). Column meta mapping appears to be stateful. 2. How would I update my recipe to have column level terms prefaced with a term node (e.g.
PII.
)? I would like to avoid the user having to type that prefix, but unsure how to add it when using
add_terms
instead of just
add_term
Version: 0.10.1 Recipe config:
Copy code
meta_mapping:
      datahub.owner:
        match: ".*"
        operation: "add_owner"
        config:
          owner_type: "group"
      contains_pii:
        match: True
        operation: "add_term"
        config:
          term: "PersonalInformation.PII"
      contains_pii:
        match: False
        operation: "add_term"
        config:
          term: "<http://PersonalInformation.No|PersonalInformation.No> PII"
      tier:
        match: "Tier 1|Tier 2|Tier 3"
        operation: "add_term"
        config:
          term: "Tier.{{ $match }}"
    column_meta_mapping:
      glossary_terms:
        match: ".*"
        operation: "add_terms"
        config:
          separator: ","
📖 1
1
🔍 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
m
@early-hydrogen-27542 the default write mode in datahub is
Patch
i.e. append meta. You can switch it to
Override
mode which will replace existing one with the new one... Although i am not sure the implication with other metadata. If you find restrictions in out of the box functionality, you can write your own custom transformer. You have a lot of flexibility in what to do with your metadata
d
Thanks Steve for helping out! 🙂
e
Thanks, Steve! Also, any thoughts on the second question?
m
As i said, you can write your own transformer if you find the out of the box functionality limited.
e
Makes sense - thanks so much, always appreciate your help here.