I have a JSON file on S3 that is Gzipped. the fil...
# ask-community-for-troubleshooting
I have a JSON file on S3 that is Gzipped. the files are in folders that follow a pattern /search/year=<year>/month_<month>/day_<day>/file_<number>.gz I dont think the file connection can handle compression or a pattern and the S3 connector can’t handle JSON and I dont see an option to select the compression. Is this correct or am I missing something?
@Liren Tu could you help here please?
@Noel Gomez, you are right that currently neither the file source connector or the S3 source connector can handle this right now. We do plan to support more formats in the S3 source connector in the future. Here is the issue you can track about it: https://github.com/airbytehq/airbyte/issues/5103
Gzip compression should already be supported though. Right now S3 source supports gzip and bzip2: https://github.com/airbytehq/airbyte/issues/5110
thanks, regarding gzip, is that auto detected? not an option I see during setup
Yes it's auto-detected, based on file extension 🙂
will zip compression support added soon in file connector?
Hey @Sheshan, this issue is probably the best to follow (even though it says S3, the addition of compression support here will eventually drip down to multiple new file sources)