Hi, Is there a way to write a destination connecto...
# ask-community-for-troubleshooting
a
Hi, Is there a way to write a destination connector which takes in binary files (pdf, doc, docx, etc) from a source (for example, S3)? Because as far as I have seen, Airbyte works with record-based data such as CSV files from a source etc.
g
you may look at SFTP connector
a
Hi, I looked into that. My question specifically is that consider this scenario. My Airbyte connection chain is supposed to work like this.
S3 Bucket -> Airbyte S3 Source Connector -> Airbyte My Dest. Connector -> Uplaod data
In the S3 bucket, user can add files of different types, such as PDF, Doc, DocX, HTML, JSON, etc. My destination connector (That I have already got working via integration tests) can upload a file to the destination. The question is that whether already available S3 Source connector can just read that binary data from a S3 bucket and send it over as-is to me? or is it based on a schema because of which it can only read record-based data such as a CSV?
Also, it looks like SFTP only supports CSV and JSON files yet. For other formats, since it cannot build dynamic streams, it won't work. Am I right?