Hi, im writing ny own custom transformer and im wo...
# ingestion
a
Hi, im writing ny own custom transformer and im wondering how can i return a list of Table properties from what i ingested on the source(glue) through a custom config? Is there any class that could i can look through to return each Dataset’s Properties?
c
Hey Jensen, if you look in the
aspects
of each
DatasetSnapshot
, these should contain the table properties – you’ll find
Status
,
OwnershipClass
,
DatasetPropertiesClass
, and
SchemaMetadata
all here
a
@chilly-holiday-80781 Thanks! I tried making it and place it on the same directory with my recipe…following the datahub-ingestion cronjob /etc/recipe/glue-crawler.yaml /etc/recipe/custom_transformer.py and when i put our custom transformer on the recipe,
Copy code
transformers:
  - type: "custom_transformer.AddOwner"
but there’s an error.
Copy code
custom-transformer is not found
this error keeps coming up….do i have to add something else on the entry_point or ?
c
Hmm, is custom_transformer either a Python file in the same directory or an installed module?
a
Copy code
ModuleNotFoundError: No module named 'custom_transformer'
yea sorry it’s a python file….and its not an installed module….but i COPY the custom transformer file with my recipe in a dockerfile with datahub-ingestion:v0.8.8 as my dockerfile base
@chilly-holiday-80781 something like this
c
Ah, so the class in the type should match the file/class name – could you give this config a shot?
Copy code
transformers:
  - type: "add_table_description.<your_class>"
a
after adding /etc/recipe to pythonpath for me it worked…haha but i’m going through some other problem in the transformer right now! Thanks for the help tho^^
c
Nice, if you’re still stuck please let us know!