hey team, i’ve seen many users here request some variation of wanting to add another table to an existing connector that uses CDC.
I’m aware of the strict configurations due to the pointer on the binlogs being applied to the entire connector.
I imagine there’s a manual process to add the new table in a seperate connector (full refresh), then update the existing connector VIA the api to add this new table without resetting the connector. (And probably updating the database).
and the continuing on with existing connector syncing.
I’d love if there’s a fullproof step by step for this detailed somewhere.
separately, has it been considered to create pointers for each table in a connector such that none of them are impacted when others are added/removed?
12/17/2021, 3:30 PM
Hi @gunu do you mean you'd like to add a new table to an already existing connection? I hope not answering the wrong question, but it's not currently supported by Airbyte. There's maybe a subtle difference for CDC I'm not aware of.
12/17/2021, 3:33 PM
I think @[DEPRECATED] Marcos Marx gave you a better suggestion than mine in your message on #dev
@gunu, this sounds like a good alternative. However, we have not tried this out yet.
For databases, this quarter (Q4 2021) we are focusing on non-CDC connections, and trying to make they much more robust and performant than they are now.
We will look into improving CDC connections + schema evolution next quarter (Q1 2022). Given the limited bandwidth, I’m afraid that we won’t test any advanced CDC idea until then. But we will definitely take into account what you have suggested here.