Hi. I have a question about scaling. I’m reading https://docs.airbyte.com/operator-guides/scaling-airbyte and see

As a reference point, the typical Airbyte user has 5 - 20 connectors and 10 - 100 connections configured. Almost all of these connections are scheduled, either hourly or daily, resulting in at most 100 concurrent jobs.

I would like to use the Postgres CDC source connector, but our source DB has over 550 tables to sync. Would Airbyte have any issues with this. We plan to deploy OSS Airbyte to Kubernetes.

Feedback has been sent to Harvestr

Sometimes users split their large table into individual connections to have better state control. Per-stream feature is almost done and will solve this problem though. I’d strongly recommend you to test using a small -> med -> large table to see how the performance works.

I see. The downside is each Postgres CDC connection requires another publication/replication slot to be created? How does the per-stream feature relate to the number of db publication slots that need to be created? Where can I read more about the per-stream feature?