Hi all! Now datahub is integrated well with airflo...
# ingestiond
damp-minister-31834
11/29/2021, 11:53 AMHi all! Now datahub is integrated well with airflow. However, my company use dolphinscheduler not airflow. So I want to ask about the approximate steps to integrate datahub with a new scheduling framework.
g
green-football-43791
11/29/2021, 6:14 PMHey there @damp-minister-31834 - we would likely need a contribution from the community to build a source to integrate with dolphinscheduler. Is this something you would be interested in contributing? It would need to query dokphinscheduler’s API and turn that metadata into datahub objects
d
damp-minister-31834
11/30/2021, 1:44 AMYear, I would be interesting. But now I don't know what I should do for the first step.😅
g
green-football-43791
11/30/2021, 1:44 AMSure, I can help with that
green-football-43791
11/30/2021, 1:44 AMYou’ll first want to figure out how you can get the metadata out of dolphin scheduler
green-football-43791
11/30/2021, 1:45 AMdoes it have an API you would hit?
green-football-43791
11/30/2021, 1:45 AMor would you parse some files?
green-football-43791
11/30/2021, 1:45 AMafter that, I can refer you to another example ingestion source that is similar to what you find out about dolphinscheduler
green-football-43791
11/30/2021, 1:45 AMand you can use that as a starting point
d
damp-minister-31834
11/30/2021, 1:55 AMYear, it offers rest api.
g
green-football-43791
11/30/2021, 1:56 AMok, you can refer to the superset source as an example of how to write a source off a rest API:
green-football-43791
11/30/2021, 1:56 AMd
damp-minister-31834
11/30/2021, 1:57 AMOk, I'll see the code first!
damp-minister-31834
11/30/2021, 2:52 AMNow I roughly know how to get the http response via rest api. But I hava a question, dolphin scheduler is a workflow scheduler flatform like airflow not a data source. So what should I do if I want to display the pipline and lineage in datahub by integrating with dolphinscheduler. I see the introduce "`Running Airflow locally with DataHub`" , there is something changed in airflow site not datahub site?
g
green-football-43791
11/30/2021, 2:58 AMgreen-football-43791
11/30/2021, 2:58 AMYou might want to take a look at this
d
damp-minister-31834
11/30/2021, 2:59 AMOk, I'll see it!
g
green-football-43791
11/30/2021, 3:00 AMit show how you can construct datajob and dataflow objects which are how datahub can parse your tasks and pipelines
d
damp-minister-31834
11/30/2021, 3:00 AMGood!
2 Views