Hey all - I am having trouble executing ingestion ...
# ingestionc
creamy-ram-28134
05/25/2023, 7:56 PMHey all - I am having trouble executing ingestion - can someone share examples for csv and file ingestion
g
gentle-hamburger-31302
05/30/2023, 7:46 AMHi @creamy-ram-28134
Which source are you using ? S3 source supports local csv files
c
creamy-ram-28134
05/30/2023, 2:12 PMSource is csv and file - deployed on kubernetes
g
gentle-hamburger-31302
06/07/2023, 11:07 AM@creamy-ram-28134 Did you tried s3 source ?
c
creamy-ram-28134
06/08/2023, 3:23 PMno
f
flaky-eve-99734
10/17/2023, 4:47 PM@gentle-hamburger-31302, we do not have s3, and can’t find an example of just local file ingestion.
a
aloof-gpu-11378
10/18/2023, 9:03 AM@dazzling-judge-80093 might help you
aloof-gpu-11378
10/18/2023, 12:11 PM@flaky-eve-99734 the source should support local files, you just need to specify the path:
like
Copy code
path_specs:
- include: "/my-folder/my-folder2/{table}/{partition_key[0]}/{partition_key[1]}/{partition_key[2]}/*.json"r
rich-barista-93413
10/23/2023, 8:08 PMIt'll be the same as normal s3 usage, but the paths will just be local filesystem paths instead of
s3://c
creamy-ram-28134
10/24/2023, 2:17 PMHey - this is what worked for me - source:
type: s3
config:
path_specs:
-
include: '/home/gopikab/gmps/gmps/VIMCHG/*.csv' But I had to mount a volume with the data to the actions pod