I have an issue,
The Devops team just reach out to me about airbyte causing production database to be slowing down.
Here is my setup.
1. Airbyte have read only access to the prod dB.
2. One slot/publication was allocated to the pipelines
3. Sync time interl set to 5 minutes
4. Sync mode is CDC
The production dB have over 80 tables.
Less than ten tables have record more than 100k, though two tables exceed 1m records.
How do effectively manage airbyte so as not to impact the production dB.
s
Sunny Hashmi (Airbyte)
12/06/2022, 9:24 PM
Hey @Qamarudeen Muhammad, can you provide more information on which database you are connecting to, and which connectors and connector versions are you using?
There is some general information on CDC here, once I know what kind of database you're using I can see if there is specific instructions to optimize.
q
Qamarudeen Muhammad
12/07/2022, 5:34 AM
I am reading from RDS Postgres 13 with Airbyte version 0.40.14.
Though on further examination, it turns out that Airbyte is not the cause of the issue on the Database.
Because, when we isolated Airbyte on the database, the issue persist.
But, I need general information on Resource usage by airbyte during cdc operation.
s
Sunny Hashmi (Airbyte)
12/07/2022, 8:54 PM
Great to hear you are narrowing down the cause, good luck with the investigation!
There is some info published in the source-postgres connector doc, although we are still in the process of benchmarking our connectors so it is not complete: https://docs.airbyte.com/integrations/sources/postgres#cdc-considerations
Although we are still collecting data to benchmark, you are welcome to follow along and leave a thumbs up on this github issue: https://github.com/airbytehq/airbyte/issues/11719