Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Is there a way to make Airbyte spawn more `s3 writers` ?

Currently it looks like it only creates `1` `writer` , and for very large tables (`syncing from pg`), this is quite slow.

So ideally you should only sync tables relatively small in size with airbyte.

Which is fine I guess because you can truncate the data from the source and sync and append.

I guess I was just in a situation now where I needed to move large amounts of data, as a one time thing.

And ones that is moved, I would truncate the table, and the sync would go faster each time, since it does not need to take all historical data.

But I think it would be nice if there could be a distribution element to the read/write.