Hi folks, I have many connections all running a custom DBT transformation AFTER it does the regular incremental sync + stock normalisation. So: 1. Pipeline with incremental sync 2. DBT Normalisation post sync 3. DBT custom run post sync When the DBT custom normalisation in step 3 fails, airbyte will retry the entire connection 3 times. However, it also re-does all of step 1, essentially re-syncing all the data and eating 3x the warehouse credits. The cause if the issue is often when the profiles file gets moved (running in docker requires it to be copied to the mount point and its not flawless), or if we introduced bad code. Is there any way these can be split - because item 1 was successful and it seems completely redundant that the entire thing is redone

Based on the information provided in the extracts, it seems that Airbyte currently does not have a built-in mechanism to split the steps in the way you described. When a failure occurs, Airbyte retries the entire connection, including the data sync and normalization steps, which can indeed consume more resources. However, there's a discussion in a [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1670321195748919) about a similar issue where a user suggested a feature request for a retry mechanism that ensures incomplete writing does not produce partial data. The support team encouraged the user to open a feature request for this. You might want to consider doing the same for your specific case. You can create a feature request by going to the [issues tab in the Airbyte repository](https://github.com/airbytehq/airbyte/issues). In the meantime, you might want to ensure that your custom DBT transformations and profiles file are correctly set up to avoid failures. The [Airbyte documentation](https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte) provides guidance on how to set up transformations with Airbyte, including how to add custom DBT transformations.