Hey team! Since I’m not a java developer I won’t submit a PR for this myself, but I noticed something which I think really makes your lives a lot harder than it should. For a few destinations, we write simple files. All of these destinations currently implement the same code for all the file formats that are supported (see, for instance, https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/d[…]tination/azure_blob_storage/writer/ProductionWriterFactory.java) The downside of this approach is that even though we have (for instance) parquet output for S3, we don’t have it for azure blob storage. even though all the code is already written to support this IMO it would save a lot of effort if there were a single “remote file destination” that handles all the type conversions, the schema, etc, and then let S3, blob, ftp, … deal with the interface to the storage itself.

Feedback has been sent to Harvestr

Good question, and yes 🙂 I’ve got a few sources that use (S)FTP for the data transport, so I thought to implement this myself. In doing so (again, no Java developer, more prolific in Python) I quickly realised this would mean implementing a parquet reader, a csv reader, a json reader, an avro reader, etc.

Hi @Huib, I think I found a GitHub issue pretty similar to the valuable feedback you shared. Feel free to thumb it up, share your opinion and discoveries on it and of course subscribe to receive updates 😄