soma chandra sekhar attaluri
11/27/2025, 3:53 AMkapa.ai
11/27/2025, 3:54 AMRESOURCE_EXHAUSTED: grpc: received message larger than max, and support notes that Airbyte only supported ~**6,000 tables** in that context. [[Oracle schema error](https://discuss.airbyte.io/t/1811)] - Another thread documents that trying to load more than ~6,000 tables can cause resource‑exhausted errors when rendering or discovering tables. [[Tables per connection](https://discuss.airbyte.io/t/84)] - With ~**300+ tables**, data appeared in Snowflake only when syncing 1–20 tables; support suggested this might be due to message size limits and recommended splitting tables across multiple connections. [[Redshift→Snowflake issue](https://discuss.airbyte.io/t/2119)] - For very many connections/tables, resource overhead becomes significant and you may need more CPU/RAM and to tune job resources. [[Resource usage](https://discuss.airbyte.io/t/995)] In your case, where only the first ~25 tables sync and anything above that “does not get started,” the most likely causes based on these sources are: 1. Message/catalog size limits (too many tables/columns in one connection). 2. Resource exhaustion (CPU/RAM) when starting larger syncs. How to mitigate/rectify: 1. Split tables into multiple connections This is the primary workaround recommended by Airbyte support in several threads when many tables cause failures or empty outputs. [[Redshift→Snowflake issue](https://discuss.airbyte.io/t/2119); [MySQL catalog size](https://discuss.airbyte.io/t/2656)] 2. Reduce per‑connection complexity - Avoid selecting all columns if not needed (fewer fields → smaller messages). [[Cloud limits](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/understand-airbyte-cloud-limits)] - Consider excluding very wide or rarely used tables from a given connection. 3. Scale and tune resources (Self‑hosted) - On self‑hosted, increase CPU/RAM and adjust job resource env vars like JOB_MAIN_CONTAINER_CPU_LIMIT, JOB_MAIN_CONTAINER_MEMORY_LIMIT to handle more work per sync. [[Resource usage](https://discuss.airbyte.io/t/995)] 4. Check connector and Airbyte versions - Some “sudden failure” or “never starts” behaviors have been improved in newer connector and platform versions; support often recommends upgrading both Airbyte and the connectors. [[MySQL→BigQuery failures](https://discuss.airbyte.io/t/2656)] If you share which deployment (Cloud vs self‑hosted), which source/destination, and roughly how many tables/columns are in those 25+ tables, you can often decide exactly how to split connections (for example, group by schema or by table size).soma chandra sekhar attaluri
11/27/2025, 3:55 AMkapa.ai
11/27/2025, 3:55 AMkapa.ai
11/27/2025, 3:56 AM