https://datahubproject.io logo
Join Slack
Powered by
Hi team How to create custom source for ingesting ...
# all-things-deployment
c
Hi team How to create custom source for ingesting our lineage file ?
o
Hi! Please see our documentation on using custom ingestion sources here: https://datahubproject.io/docs/how/add-custom-ingestion-source/
c
I tried using that but it's not working
Getting this error while creating the python package for source class
o
Do you have file create permissions in that directory on this machine? Permission Denied is an OS level error. You may need to chmod the directory to give proper permissions
c
I did Chmod -R 777 src
Still getting this
??
o
That's still an operating system level error. You will need to change the ownership from what it currently is (most likely root from someone using sudo) to a user group you have access to.
c
Using Sudo with this command to change permissions
It is asking for pwd
Any idea ?
o
I recommend you get in touch with your internal IT department to get the required credentials/set up the machine in a way that you have the proper access needed.
c
Hi I am able to install the custom source package and now. I am trying to ingest my recipe file but I am getting this error Nameerror: name intialize is not defined
Can you help me ?
o
Looks like maybe a spelling error of some sort (
intialize
vs 
initialize
)? What command was executed to get this error?
c
I am running datahub ingest -c recipe.yml
o
Is this using a custom source? 
site-packages/edccustomsource
The callstack is mostly unreadable in the picture so it's hard to see where the name error originates from, but it doesn't seem to be within one of DataHub's Python libraries.
c
I have created edccustomsource python package and installed at my image And when I am trying to run my recipe it's giving key error as no module found -edccustomsource
What I will give in source type?
o
As this is a custom source, it's hard to know what is going wrong from our side without knowing how the custom source is written. It seems like there is probably an issue inside that custom source.
c
I am using the same custom ingestion py file that has been mentioned in example https://github.com/acryldata/meta-world/tree/master/custom_sources/src/my-source And using This and setup file I am creating edccustomsource package that I have installed But it is giving key error in source type
o
You have not modified the file at all besides its name and it fails with that error? Or you have written custom Python code on top of what that file does and it runs into that error?
Open in Slack
PreviousNext
Powered by