<@U04S8PLJWMT> is there any limit to the number of...
# ask-ai
s
@kapa.ai is there any limit to the number of tables transfered from a database because when i am trying to make a sync upto 25 tables are getting synced more than that are not getting started why is that how to rectify this error
k
Yes, there are practical limits related to how many tables/streams you can sync in a single Airbyte connection, and hitting those limits can cause syncs not to start or to behave inconsistently. From the provided sources: - Airbyte Cloud has a hard limit of 1,000 streams per connection and per discover call, and 20,000 fields per connection. [[Cloud limits](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/understand-airbyte-cloud-limits)] - Even below those numbers, users have hit issues when syncing large numbers of tables: - Oracle schema with 13,541 tables fails with
RESOURCE_EXHAUSTED: grpc: received message larger than max
, and support notes that Airbyte only supported ~**6,000 tables** in that context. [[Oracle schema error](https://discuss.airbyte.io/t/1811)] - Another thread documents that trying to load more than ~6,000 tables can cause resource‑exhausted errors when rendering or discovering tables. [[Tables per connection](https://discuss.airbyte.io/t/84)] - With ~**300+ tables**, data appeared in Snowflake only when syncing 1–20 tables; support suggested this might be due to message size limits and recommended splitting tables across multiple connections. [[Redshift→Snowflake issue](https://discuss.airbyte.io/t/2119)] - For very many connections/tables, resource overhead becomes significant and you may need more CPU/RAM and to tune job resources. [[Resource usage](https://discuss.airbyte.io/t/995)] In your case, where only the first ~25 tables sync and anything above that “does not get started,” the most likely causes based on these sources are: 1. Message/catalog size limits (too many tables/columns in one connection). 2. Resource exhaustion (CPU/RAM) when starting larger syncs. How to mitigate/rectify: 1. Split tables into multiple connections This is the primary workaround recommended by Airbyte support in several threads when many tables cause failures or empty outputs. [[Redshift→Snowflake issue](https://discuss.airbyte.io/t/2119); [MySQL catalog size](https://discuss.airbyte.io/t/2656)] 2. Reduce per‑connection complexity - Avoid selecting all columns if not needed (fewer fields → smaller messages). [[Cloud limits](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/understand-airbyte-cloud-limits)] - Consider excluding very wide or rarely used tables from a given connection. 3. Scale and tune resources (Self‑hosted) - On self‑hosted, increase CPU/RAM and adjust job resource env vars like
JOB_MAIN_CONTAINER_CPU_LIMIT
,
JOB_MAIN_CONTAINER_MEMORY_LIMIT
to handle more work per sync. [[Resource usage](https://discuss.airbyte.io/t/995)] 4. Check connector and Airbyte versions - Some “sudden failure” or “never starts” behaviors have been improved in newer connector and platform versions; support often recommends upgrading both Airbyte and the connectors. [[MySQL→BigQuery failures](https://discuss.airbyte.io/t/2656)] If you share which deployment (Cloud vs self‑hosted), which source/destination, and roughly how many tables/columns are in those 25+ tables, you can often decide exactly how to split connections (for example, group by schema or by table size).
s
@kapa.ai i am talking about the open source version that is available not cloud
k
A message from kapa.ai
New message text here