Hi, is it possible to ingest data from hdfs datab...
# ingestionc
cool-iron-6335
07/12/2021, 9:46 AMHi, is it possible to ingest data from hdfs database ? If it is, can you give me a recipe example ? The metadata ingestion document seems not to mention to this aspect.
b
big-carpet-38439
07/12/2021, 1:02 PMIs this a Hive DB?
c
cool-iron-6335
07/12/2021, 1:51 PM@big-carpet-38439 This is a hadoop database.Hive Database has already been documented
g
gray-shoe-75895
07/12/2021, 5:00 PM@cool-iron-6335 I was under the impression that HDFS usually stores raw files - what exactly are you looking to ingest?
gray-shoe-75895
07/12/2021, 5:01 PMGenerally we haven’t yet built out filesystem-based ingestion e.g. for S3 ingestion, we rely on glue crawlers https://datahubproject.io/docs/metadata-ingestion/s3-ingestion/
➕ 1
c
cool-iron-6335
07/13/2021, 2:05 AMCan i ingest raw files from HDFS ?
g
gray-shoe-75895
07/13/2021, 2:08 AMNot at the moment - out of curiosity, what sort of files do you have stored in HDFS
d
delightful-orange-22738
01/28/2022, 12:22 PMhow can i create hdfs platform?
i seen this in demo datahub https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:hdfs,SampleHdfsDataset,PROD)/Schema?is_lineage_mode=false
I need same thing
m
microscopic-mechanic-13766
08/17/2022, 10:46 AMHello, are there any news regarding ingestion on HDFS either as a source or of the files contained in it??
Thanks in advance!
d
delightful-orange-22738
11/25/2022, 10:50 AM@microscopic-mechanic-13766 i wan’t need what in files.
i was need see only fact of using this folder in airflow task
2 Views