Hello everyone,
We're conducting a POC using the Airbyte Cloud trial version. I am trying to set up a connection between an MS SQL Server (source) and Azure Data Lake Storage (ADLS) Blob (target).
The goal is to split files before writing them to the Blob by configuring the Azure Blob Storage file spill size with the desired MBs. Additionally, we configured the Azure Blob Storage output buffer size to match the spill size to ensure that files are written only once to the Blob.
However, despite these efforts, Airbyte creates the file in the Blob as 0 bytes initially, and the file size keeps increasing as the process progresses until completion. This behavior creates an issue because we have an Event Grid that triggers a downstream task as soon as the file is created.
We are looking for a solution to ensure that Airbyte writes the file simultaneously as it is created in the Blob and does not continue appending rows to the same file.
Note: We cannot use an intermediate Blob storage before pushing the data to the target Blob (which is linked to the Event Grid).
Is there any way to solve this issue?