What’s the ETA on allowing new source tables in a ...
# advice-data-ingestionr
Rocky Appiah
08/12/2022, 12:38 PMWhat’s the ETA on allowing new source tables in a PG instance to be added without having to refresh the entire destination? Combined with the throughput on large tables, this starts to become a problem 😞
m
Marcos Marx (Airbyte)
08/12/2022, 12:45 PMThis is a work in progress @Rocky Appiah but very complex one 😞 had a lot of changes in Airbyte protocol
r
Rocky Appiah
08/12/2022, 1:29 PMOk so no ETA at the moment?
Rocky Appiah
08/12/2022, 1:30 PMAny recommendations on workarounds until then?
m
Marcos Marx (Airbyte)
08/12/2022, 1:56 PMNo ETA unfortunately, recommendation is separate big tables in single connections and small tables in a generic one
r
Rocky Appiah
08/12/2022, 3:56 PM@Marcos Marx (Airbyte): this won’t break capturing incremental updates? i.e.:
Incremental | Deduped + historyRocky Appiah
08/12/2022, 3:56 PMOr do we need a separate slot for each connection, I assume so 😞
m
Marcos Marx (Airbyte)
08/12/2022, 4:04 PMIf you break into multiple connection the best practice is to have individual slots
r
Rocky Appiah
08/12/2022, 5:41 PM@Marcos Marx (Airbyte): can I use the same publications for all tables, just different replication slots yes?
2 Views