Hi all! Now datahub is integrated well with airflo...
# ingestion
d
Hi all! Now datahub is integrated well with airflow. However, my company use dolphinscheduler not airflow. So I want to ask about the approximate steps to integrate datahub with a new scheduling framework.
g
Hey there @damp-minister-31834 - we would likely need a contribution from the community to build a source to integrate with dolphinscheduler. Is this something you would be interested in contributing? It would need to query dokphinscheduler’s API and turn that metadata into datahub objects
d
Year, I would be interesting. But now I don't know what I should do for the first step.😅
g
Sure, I can help with that
You’ll first want to figure out how you can get the metadata out of dolphin scheduler
does it have an API you would hit?
or would you parse some files?
after that, I can refer you to another example ingestion source that is similar to what you find out about dolphinscheduler
and you can use that as a starting point
d
Year, it offers rest api.
g
ok, you can refer to the superset source as an example of how to write a source off a rest API:
d
Ok, I'll see the code first!
Now I roughly know how to get the http response via rest api. But I hava a question, dolphin scheduler is a workflow scheduler flatform like airflow not a data source. So what should I do if I want to display the pipline and lineage in datahub by integrating with dolphinscheduler. I see the introduce "`Running Airflow locally with DataHub`" , there is something changed in airflow site not datahub site?
You might want to take a look at this
d
Ok, I'll see it!
g
it show how you can construct datajob and dataflow objects which are how datahub can parse your tasks and pipelines
d
Good!