Hey everyone, I have a use case where I want to se...
# ingestion
b
Hey everyone, I have a use case where I want to set some dataset properties and owners based on elements parsed from the description. I want to create a custom transformer to do this and read through the documentation on how to do so, but the example provided is for setting aspect value from an outside source. What I want to know is, is it possible to set an aspects value using the value of another aspect at ingestion time? Based on the documentation, since the _aspect_name()_ function returns just a single string, it seems like the answer is no unless I can somehow pull and set that information during the __init__() or create() functions. I found the following threads which seem to be related as well: 1. https://datahubspace.slack.com/archives/CUMUWQU66/p1655191762153639 2. https://datahubspace.slack.com/archives/CUMUWQU66/p1655796024377539 I think my use case is closest to sibren's (2), but Thomas Larsson (1) talks about the issue I have of a transformer needing multiple aspects to perform it's job. I saw Chris suggested making API calls to datahub (or I presume to datahub), but if the dataset doesn't already exist in datahub, would that be possible?
m
Hey Ryan, in your case, do you want to listen to aspect A and make changes to aspect B? The current transformer implementation follows a
SingleAspectTransformer
pattern here. One could conceivably add a
MultiAspectTransformer
that declares what aspects it wants to listen to and the
BaseTransformer
could take that into account and call it appropriately. This would be a useful addition to the framework. cc @wonderful-quill-11255 if you are interested in working on this as well.
b
Thanks for the reply Shirshanka, this is helpful. I can see about possibly making this addition at some point. Appreciate the idea!