https://linen.dev logo
#feedback-and-requests
Title
# feedback-and-requests
r

Remi Salmon

12/01/2021, 4:41 PM
Hi all, I am curious if the following trick if doable with Airbyte: we are currently syncing source A to destination B using <not Airbyte> and would like to switch to Airbyte to sync A to destination C, however we are not able to complete a full sync of A using Airbyte when the connection is first run (A is a Heroku postgres db and a full sync throws errors ending in Heroku killing the connection, B and C are Snowflake dbs). Is it possible to trick Airbyte by copying the existing data from B to C and having Airbyte start with incremental syncs from A to C using from the last cursor available from the data in C, instead of starting with a full sync? Hope that all makes sense.
u

user

12/01/2021, 6:23 PM
No because A => C is (Heroku to Snowflake) and connection B => C (Snowflake to Snowflake) are different connectors. Are you having problem when you try to sync A to B together with A to C, or only A to C (Airbyte) is having problem to finish the sync?
c

Christopher Wu

12/01/2021, 7:17 PM
The disconnection issue happens when syncing A to C using only Airbyte, no other operations are going on in A but Heroku fails to perform health checks on A after some time (4 hours) and end up killing the connection (
Exception in thread "main" java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
is what we see in the Airbyte log)
u

user

12/02/2021, 12:02 AM
Maybe one suggestion: create a view to limit the import using incremental, then after the sync you can move the window the view has access. Example: create view example (select * from table where date >= 2010 and date <= 2011), after the first sync move the window to date >= 2011 and date <= 2012
2 Views