Hi, im writing ny own custom transformer and im wondering how can i return a list of Table properties from what i ingested on the source(glue) through a custom config? Is there any class that could i can look through to return each Dataset’s Properties?

Hey Jensen, if you look in the

aspects

of each

DatasetSnapshot

, these should contain the table properties – you’ll find

Status

,

OwnershipClass

,

DatasetPropertiesClass

, and

SchemaMetadata

all here

@chilly-holiday-80781 Thanks! I tried making it and place it on the same directory with my recipe…following the datahub-ingestion cronjob /etc/recipe/glue-crawler.yaml /etc/recipe/custom_transformer.py and when i put our custom transformer on the recipe,

transformers:
  - type: "custom_transformer.AddOwner"

but there’s an error.

custom-transformer is not found

this error keeps coming up….do i have to add something else on the entry_point or ?

Hmm, is custom_transformer either a Python file in the same directory or an installed module?

Ah, so the class in the type should match the file/class name – could you give this config a shot?

transformers:
  - type: "add_table_description.<your_class>"

after adding /etc/recipe to pythonpath for me it worked…haha but i’m going through some other problem in the transformer right now! Thanks for the help tho^^

Nice, if you’re still stuck please let us know!