I am Prototyping a project using open source Airbyte to sync about 100 tables in a self manage cloud SQL Server to Bigquery. For context, some of these tables are smaller and don’t get updated often, but others have millions of rows with 100,000+ rows written per day. Because of how Airbyte likes to do a “Reset all streams” for any edit to a connection setup in Airbyte… Should I make a single connection for each table, so that we can make changes later and not reset every table? Or maybe I am thinking about this the wrong way?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2711 to ensure timely resolution!

Hi. You should not reset the data for every small change you make - dont see the point unless this is 100% crucial. you should not also make 1 connection for 1 table or else you are going to have hundreds of connection which will be very hard to manage. you can use airbyte api to update sources, destinations connections etc.. which does not require to make those resets at all.

Understood. Correct me if I am wrong, but doesn't this ticket suggest the reset will be forced to happen, particularly when using MSSQL as the source? https://github.com/airbytehq/airbyte/issues/13781

That ticket is to be able to reset a specific table if needed upon change instead the whole source which in some cases is very time consuming

There may be some distinction there I am missing. However, that sounds like the same thing to me🤔. let's say I have all the tables setup to sync via a single connection, and then I add a column I want to import to one of those tables. I would then need to reset all data for each table to get the new column, correct?

in general yes, though no- you can update the connection - reset only the specific table - update to previous state with the new column- this is a workaround

I see. Thanks for the extra details and quick responses. Much appreciated! I will test out the workaround you describe

Hey @Grant Pendrey, were you able to get this working? Let us know and feel free to post more questions in the channel!