Hi Team, I'm new to Airbyte. Currently we use Airflow (Google Cloud Composer) SFTPtoGCS operator to pull raw files (Parquet, Avro, txt, csv, json) from SFTP locations to GCS (Google Cloud Storage), and later load into BigQuery (SFTP->GCS->BigQuery) based on schema/ data dictionary; but checking if Airbyte can do SFTP->GCS. I tried using

File

source with SFTP and

Google Cloud Storage

as destination within Airbyte, but it requires for me to 1. Specify File schema Reader Options for each file 2. Create a source for each file I am checking if Airbyte can get all raw files from SFTP and dump into GCS location on scheduled bases without supplying reader option/ schema.

Today the File connector is very limited and only allow you to get one file, see issue. Maybe you can expand the File connector to a custom connector that can handle your use case (my suggestion)

Thanks a lot @[DEPRECATED] Marcos Marx.