Hi all, I am curious if the following trick if doa...
# feedback-and-requests
r
Hi all, I am curious if the following trick if doable with Airbyte: we are currently syncing source A to destination B using <not Airbyte> and would like to switch to Airbyte to sync A to destination C, however we are not able to complete a full sync of A using Airbyte when the connection is first run (A is a Heroku postgres db and a full sync throws errors ending in Heroku killing the connection, B and C are Snowflake dbs). Is it possible to trick Airbyte by copying the existing data from B to C and having Airbyte start with incremental syncs from A to C using from the last cursor available from the data in C, instead of starting with a full sync? Hope that all makes sense.
u
No because A => C is (Heroku to Snowflake) and connection B => C (Snowflake to Snowflake) are different connectors. Are you having problem when you try to sync A to B together with A to C, or only A to C (Airbyte) is having problem to finish the sync?
c
The disconnection issue happens when syncing A to C using only Airbyte, no other operations are going on in A but Heroku fails to perform health checks on A after some time (4 hours) and end up killing the connection (
Exception in thread "main" java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
is what we see in the Airbyte log)
u
Maybe one suggestion: create a view to limit the import using incremental, then after the sync you can move the window the view has access. Example: create view example (select * from table where date >= 2010 and date <= 2011), after the first sync move the window to date >= 2011 and date <= 2012