Join Slack
Powered by
Second question today: How can intest HDFS in Hado...
# ingestion
s
sparse-barista-40860
07/18/2022, 7:52 PM
Second question today: How can intest HDFS in Hadoop?
g
gentle-camera-33498
07/19/2022, 12:58 AM
Do you mean ingest HDFS metadata into DataHub? The DataHub has a metadata modeling that is fixed but applies to many use cases. As I don't believe there is an ingestion already implemented by the community, you can implement your own. I'll give you the documentation links here. You can see the available sources here and a detailed introduction to the ingestion process:
https://datahubproject.io/docs/metadata-ingestion
You can use the DataHub SDK to implement your own Metadata Emitters:
https://datahubproject.io/docs/metadata-ingestion#using-as-a-library-sdk
You can even implement your custom source and your own recipe config:
https://datahubproject.io/docs/how/add-custom-ingestion-source
4
Views
Open in Slack
Previous
Next