Hello. We are using Airbyte for replicating Aurora to Postgres RDS with about 2T rows and 2TB database. We have barely processed 200M rows in 24 hours. At this rate we are looking at 10-15 days for the initial sync. Is there anyway the initial full refresh can be accelerated?

Hi @Vikram Kumar, I think the bottleneck you have is related to the default

fetchsize

Airbyte uses to read the source in batch. This

fetchsize

is today of 10000 records. This is not configurable at the moment and if you want to customize this you'll have to build Airbyte your own version of the Postgres connector and patch this file

airbyte-db/lib/src/main/java/io/airbyte/db/jdbc/PostgresJdbcStreamingQueryConfiguration.java

. An issue exists in our repo about this topic and we plan to make the fetch size configurable.

Thanks for the reply. What’s a reliable way to pause the sync, stop the docker, reset those configs and resume?

We do not have a pause feature, but if you're running incremental sync, stopping the sync job and restarting will make the new sync start from the last checkpointed cursor by the previous job.