https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
b

Bhaumik Shah

01/12/2022, 7:36 PM
Hi Team, I'm new to Airbyte. Currently we use Airflow (Google Cloud Composer) SFTPtoGCS operator to pull raw files (Parquet, Avro, txt, csv, json) from SFTP locations to GCS (Google Cloud Storage), and later load into BigQuery (SFTP->GCS->BigQuery) based on schema/ data dictionary; but checking if Airbyte can do SFTP->GCS. I tried using
File
source with SFTP and
Google Cloud Storage
as destination within Airbyte, but it requires for me to 1. Specify File schema Reader Options for each file 2. Create a source for each file I am checking if Airbyte can get all raw files from SFTP and dump into GCS location on scheduled bases without supplying reader option/ schema.
1
u

[DEPRECATED] Marcos Marx

01/12/2022, 11:28 PM
Today the File connector is very limited and only allow you to get one file, see issue. Maybe you can expand the File connector to a custom connector that can handle your use case (my suggestion)
b

Bhaumik Shah

01/13/2022, 4:17 PM
Thanks a lot @[DEPRECATED] Marcos Marx.
4 Views