Hello Airbyte Team, I am currently setting up an Airbyte PostgreSQL source connector using CDC for a large PostgreSQL database and am encountering an issue during source discovery. The database contains around 17,000+ tables. For most of these tables, I have enabled CDC, while a few are configured with Full Refresh. During the initial setup, I created a replication slot and assigned 1,000 tables to a publication, but the discovery process failed. I then tried reducing the number of tables and tested again with only 10 tables in the publication, but the same issue still occurred. I also created a fresh connection, but the problem persists. Even when the publication contains only 10 tables, the issue continues, which makes me think this should not be related to payload size. To troubleshoot further, I: • Increased

MICRONAUT_SERVER_MAX_REQUEST_SIZE

to 500 MB However, the issue still remains. I also have a question regarding CDC configuration: Since each source connection has its own publication name, and we run an

ALTER PUBLICATION

command to add around 200 tables per publication, how will enabling CDC for ~80% of the tables in the database affect this process? My understanding is that only the tables included in the specific publication (e.g., the 200 tables) should be considered for source discovery and data replication, regardless of how many tables have CDC enabled in the database overall. Could you please confirm if this understanding is correct? Additionally, if providing log dumps or diagnostic information would help identify the root cause of this issue, please let me know the steps required to collect and share the relevant logs

The discover process isn't scoped to your publication, so it is still attempting to fetch column-level metadata for every table on your server. I replied to your previous comment here and never saw a response: https://airbytehq.slack.com/archives/C021JANJ6TY/p1772229396137299?thread_ts=1771935040.264669&cid=C021JANJ6TY

Hi Matt, All these tables are under one schema, and I have added only a few tables to the publication. So only those tables’ metadata (column details) would be used by the Airbyte process, right? You mentioned this is a limitation. Is there any way I can overcome this? I have also tried creating connections with just 10 tables per connection, but it still failed.

The discover process isn't scoped to the publication. A publication is only applicable to CDC, and a sync can contain tables using full refresh. The discover process must return metadata for both of these use cases. The easiest way to shrink the runtime and payload returned by discover is to separate tables into multiple schemas. If that isn't an option, you'd require the regex feature I mentioned in the other thread, which isn't currently scheduled for delivery.

ok thanks for your info, so i cant procced with this its a prod db so we cant move the tables into other schema