Hi, team. We are running into an issue with a MySQL -> Snowflake sync on Airbyte

2.0.1

and would appreciate any help. Context • Large tables, with millions of rows • The sync runs, but: ◦ the UI shows 0 records / 0 bytes ◦ data appears partially loaded in Snowflake ◦ some streams seem to be missing from state What we're seeing: • Logs show high parallelism, for example:

Table will be read by 42 concurrent partition reader(s)

• We have also seen

OOMKills

on the source and orchestrator for large runs • I tried increasing memory, but I still cannot get the orchestrator memory above

1Gi

What we have tried: • Full refresh + clear data • Incremental vs full refresh • Running one table at a time • Recreating the connection via Terraform • Increasing the instance size • Increasing job memory This issue seems relevant specifically to the orchestrator memory problem, because it describes the orchestrator ignoring configured memory and always using `1Gi`: https://github.com/airbytehq/airbyte/issues/72833 Questions • Is there a supported way in

2.0.1

to reduce partition/concurrency for large MySQL tables? • Is there a reliable way to increase orchestrator memory above

1Gi

? • Has anyone seen partial loads in Snowflake with the UI still showing 0 records / 0 bytes? Thanks in advance! FYI, @Olga Ivanchenko and @Carolina Buckler

Hi Jamie, The orchestrator memory issue may be the root cause of all three symptoms you described here. Upgrade to 2.1 and/or split large tables into separate connections as a workaround in the meantime while the team looks into this.

Hi @Henry Mattingly, thanks for the follow-up. Good news: after upgrading to 2.1.0, the orchestrator is now correctly picking up the memory configuration. On 2.0.1 it was stuck at 1Gi regardless of what we set in values.yaml, but on 2.1.0 the

JOB_MAIN_CONTAINER_MEMORY_LIMIT

from the ConfigMap is being respected. We've bumped it to 4Gi and can confirm the replication pods are launching with the correct limits. Running syncs now to verify the OOM issues are resolved. Thanks for pointing us in the right direction! FYI, @Olga Ivanchenko, @Carolina Buckler