Yusuf Mirkar
05/18/2023, 8:21 PMkapa.ai
05/18/2023, 8:21 PMkapa.ai
05/18/2023, 8:22 PMhot_standby_feedback
to true
on the replica server. This prevents the primary server from deleting write-ahead logs when the replica is busy serving user queries. However, the downside is that the write-ahead log will increase in size. [Source](https://docs.airbyte.com/integrations/sources/postgres) sql # In postgresql.conf on the replica server hot_standby_feedback = true
2. Sync data when there is no update running in the primary server, or sync data from the primary server. [Source](https://docs.airbyte.com/integrations/sources/postgres) 3. Choose an appropriate sync mode for each table within the same database. Airbyte supports several sync modes for Postgres, like full, incremental, and logical replication (also called CDC). Selecting the right sync mode for each table can help optimize the sync speed. [Source](https://airbyte.com/tutorials/postgres-replication) Please note that increasing the max_standby_archive_delay
and max_standby_streaming_delay
is not recommended, as it is usually hard to tell how much time it will take to sync all the data, and this approach is not very practical. [Source](https://docs.airbyte.com/integrations/sources/postgres)Yusuf Mirkar
05/18/2023, 8:24 PMkapa.ai
05/18/2023, 8:24 PMkapa.ai
05/18/2023, 8:24 PMJOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. This will help you allocate more memory to the source and destination containers, which can improve performance. For example, you can modify these environment variables in your .env
file or in your Kubernetes deployment configuration. Here's an example of how a user set them in a discussion thread: [link](https://discuss.airbyte.io/t/increase-sync-worker-resources/1714/3). For more information on configuring Airbyte, you can refer to the [shared configuration documentation](https://docs.airbyte.com/operator-guides/configuring-airbyte/#shared).