How does airbyte write to s3? Does it write after each file is created, or does it collect all the output from the source, and divides it into parts and then writes all at once? Asking because I wonder if the logs would show any activity when running a big download

Think it writes when it reaches a specific chunk size limit ( 10 MB i think), or when the source stream is finished, in order not to bloat the RAM of the host. But the S3 connector prints on the logs when it uploads the chunks, weird if you don’t see anything

Thanks for the response! It shows something in the log when it writes to S3, but in my case it would collect 200MB of compressed data in a JSON file before it wrote. Meanwhile nothing would happen in the logs for ~2hrs while it was running, and it looked like it was stuck hence my doubt. All clear on how it works now though