I have a JSON file on S3 that is Gzipped. the files are in folders that follow a pattern /search/year=<year>/month_<month>/day_<day>/file_<number>.gz I dont think the file connection can handle compression or a pattern and the S3 connector can’t handle JSON and I dont see an option to select the compression. Is this correct or am I missing something?

@Liren Tu could you help here please?

@Noel Gomez, you are right that currently neither the file source connector or the S3 source connector can handle this right now. We do plan to support more formats in the S3 source connector in the future. Here is the issue you can track about it: https://github.com/airbytehq/airbyte/issues/5103

thanks, regarding gzip, is that auto detected? not an option I see during setup

Yes it's auto-detected, based on file extension 🙂

will zip compression support added soon in file connector?

Hey @Sheshan, this issue is probably the best to follow (even though it says S3, the addition of compression support here will eventually drip down to multiple new file sources)