question regarding adding a pipeline in datahub. W...
# ingestions
square-activity-64562
11/09/2021, 4:53 PMquestion regarding adding a pipeline in datahub. We have a pipeline which consists of AWS Kinesis, S3, Lamdba, redis channels, python processing applications etc. Probably a few more things. I was thinking of documenting this pipeline through adding the lineage in datahub. I think the following would be required
• some way to represent redis channels
• some way to represent Kinesis streams
• some way to represent generic processors - currently only airflow seems to be shown in pipelines.
I was thinking we can model redis channels and kinesis streams as datasets. Is there any limitation currently in the model to represent generic processors - like AWS lambda or python apps? I was thinking these could be tasks in datahub.
Has anyone tried representing processing jobs outside airflow in datahub? Any feedback of how that went would be helpful.
plus1 4
h
high-hospital-85984
11/09/2021, 5:00 PMDataProcesshigh-hospital-85984
11/09/2021, 5:02 PMBut for example Kafka connectors are modeled as jobs (check the
metadata-ingestionhigh-hospital-85984
11/09/2021, 5:06 PMBut I feel it's a bit artificial to force a "flow" on that. The generic DataProcess would have been good for cases like this.
b
big-carpet-38439
11/09/2021, 6:23 PMI feel like Data Process should represent a specific instance of a Task
big-carpet-38439
11/09/2021, 6:23 PMWhich would need a flow nonetheless
big-carpet-38439
11/09/2021, 6:24 PMIt's a bit unfortunate that we've forced Data Tasks to be associated with a parent flow..
big-carpet-38439
11/09/2021, 6:24 PMOne thing we were considering is an "urn v2" which would address this issue (among others) in our urns
s
square-activity-64562
11/10/2021, 10:17 AMSeems like a feature request then. I will add it in the relevant channel for everyone's reference
2 Views