Hi all! I have the following case and I am curiou...
# ask-community-for-troubleshooting
n
Hi all! I have the following case and I am curious if Airbyte could be applied in this situation. I have a customer who has data on an SFTP server. The files on the SFTP server are encrypted with a key and are named .csv.gpg. Is there a possiblity to transfer the files, decrypt them and have S3 as a destination, using Airbyte? Currently the .gpg files cannot be picked up by Airbyte as they are not in the form of .csv / .json as described in the SFTP connector. Thanks!
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, based on the information extracts, it seems that Airbyte's SFTP connector currently supports only CSV and JSON files. You can find more details about the SFTP connector in the [documentation](https://docs.airbyte.com/integrations/sources/sftp). For your specific use case, you might need to create a custom solution or contribute a feature to Airbyte that allows handling encrypted files like .csv.gpg.
g
@Nils de Bruin You might be interested in this feature request: https://github.com/airbytehq/airbyte/issues/25786
The short answer to your question is that Airbyte is not currently suited to this use case. It assumes that Sources and Destinations can communicate via the Airbyte Protocol which requires that data be synced in a structured format (which encrypted data precludes unless it’s decrypted and parsed on the stream). You could write a custom connector, but you would probably need to write both a custom source and destination because neither the File source nor S3 destination will behave as you’d like.
n
Ok thanks for the response @Gabriel Levine!!!!
I am now investigating this by making use of rclone --> s3 and using a Lambda trigger to process the input. Not ideal, but it can do the job for now I think!
g
We do something similar with Cloud Functions for Box to GCS transfers 🙂
👍 1