hi All, I am trying to ingest a CSV file to datahu...
# troubleshoot
f
hi All, I am trying to ingest a CSV file to datahub
localhost:8080
from my local machine. Below is the yaml file I have created named as `csvingestion.dhub.yaml`:
source:
type: csv-enricher
config:
# relative path to your csv file to ingest
filename: .\path\to\file\census_income_morethan50K.csv
sink:
type: "datahub-rest"
config:
server: "<http://localhost:8080>"
I am getting the below error when I do the ingestion using the command
python3 -m datahub ingest -c .\path\to\file\csvingestion.dhub.yaml
I have also done installations using
python3 -m pip install acryl-datahub[csv]
and
python3 -m pip install acryl-datahub[csv-enricher]
Please advise what should be done. Also, if there is another effective way to ingest local CSV file please let me know. Thanks in advance!
1
b
The name of the file you attempted to ingest suggest that you are ingesting a random CSV file in the hopes of generating a dataset. But it's not the purpose of CSV enricher. It only accept CSV containing metadata of an existing entity in datahub.
CSV files you would have to programmatically generate dataset or use s3 source
g
@fast-vegetable-81275 As per doc yes
https://datahubproject.io/docs/generated/ingestion/sources/s3
Copy code
Note that because the profiling is run with PySpark, we require Spark 3.0.3 with Hadoop 3.2 to be installed