Not sure this is the right channel to ask, but I’m going to use Powered by Airbyte, orchestrated by Airflow (using the Airbyte Cloud API) and i’m wondering what would be the simplest way to : trigger syncs which sync stream in incremental refresh mode as primary mode, BUT, offer my customers to manually choose to do full refresh of some streams (for instance, to capture deletes, fix missing data, etc). I would need to create multiple connections right?

A message from kapa.ai

Yes, you're on the right track. To achieve this, you can create multiple connections for the same source and destination, with different sync modes for each stream. For the primary sync mode, you can use the Incremental sync mode, and for the manual full refresh, you can use the Full Refresh sync mode. You can refer to the following sync modes: 1. [Incremental sync](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append) - syncs only new data since the last sync. 2. [Full Refresh - Append](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-append) - retrieves all available data from the source and appends it to the destination table, creating duplicates if syncing similar information multiple times. 3. [Full Refresh - Overwrite](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite) - retrieves all available data from the source, destroys all data in the existing destination table, and then pulls the new data in. By creating separate connections with different sync modes, you can offer your customers the flexibility to choose between incremental syncs and full refreshes for specific streams. Keep in mind that Incremental sync mode does not propagate deleted records to the destination. If you need to capture deletes, you might want to consider using [Change Data Capture (CDC)](https://airbyte.com/blog/change-data-capture-definition-methods-and-benefits) replication as an alternative to incremental synchronization. For orchestrating your syncs with Airflow, you can use the Airbyte Cloud API to trigger the appropriate connection based on your customer's choice.

👋 @Loris Guignard It is possible to create a connection with incremental sync as the configuration option. That connection can be updated to full refresh through the Airbyte API as well. These configuration options are at the "connection"-level, and affect all streams within that connection. We do not yet have configuration options available to be customized at the "stream"-level yet through the API. But we will be moving towards that future this year.

Thanks! another idea I had: can we update the connection state (i.e, the latest “updated_at” of a stream for example) by API?

All customizations available for connections on creation (or update) are listed here