I am adding a custom Source by following https://datahubproject.io/docs/metadata-ingestion/adding-source. This custom source is very specific to my company. I was wondering what is the right approach to build it. I can think of 2 options. option 1: Create fork of datahubproject in my company and include the source in it and add it to setup.py, Option 2: Create a repo only for my custom sources/sinks/transform and install it on top of datahub. I prefer option 2 but not sure if its feasible how to include a plugin in setup.py

Hi @numerous-cricket-19689 take a quick look at this PR from @bland-orange-95847 (https://github.com/linkedin/datahub/pull/3324). It describes at a high level how to do this.

@numerous-cricket-19689 hope this helps you. If you need more information feel free to contact me

Hi, @bland-orange-95847, I have a question I don't understand. I've added a custom data source following the tutorial on the datahub website. I have now been able to successfully ingest data using the CLI. However, the UI method is unsuccessful, and an error message is displayed indicating that the registered custom source cannot be found. What should I do, please?

@purple-refrigerator-27989 the UI ingestion is using the DataHub actions container. To be able to use a custom source there you would need to replace that image in your deployment

@bland-orange-95847 Thank you very much, but how do I modify the datahub-actions container? I don't know which part of the code is related to UI ingestion

I mean the container built by this repo https://github.com/acryldata/datahub-actions but to be honest I am currently not sure if its possible to use a custom ingestion source by adding your python module to it because in the UI you normally select the source and its a predefined list. Would need to look it up in more detail. We extended the existing functionality, at least that worked

Thank you very much for your answer @bland-orange-95847 , If you know more details, please let me know.

@purple-refrigerator-27989 there is a

other

type to be chosen where you can fill out the YAML config for a custom source. So basically it should be possible to extend the container with your module and use it via the UI ingestion

@bland-orange-95847 Do you mean

"other"

type of the "create new Source" in UI interface? I have tried to ingest custom sources in this way, but it is unsuccessful, and an error message is displayed indicating that the registered custom source cannot be found.

yes I mean exactly this but as I wrote. For UI ingestion this container image is used https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L60 If you trigger an ingest it will download the respective python packages and execute. So for custom sources you would need to intercept that and extend functionality there

So how to intercept that and extend functionality?🥺I don't understand.

I also have not done this before but if you try to extend the actions image (extend docker image) and install your python module with custom source you may be able to use it via UI but you will need to try. I have no working example for this. Maybe @dazzling-judge-80093 can support here?