Hello Airbyte Team, We are having scaling Issues while syncing BigQuery to Clickhouse. We are using Airbyte in Docker Containers on 1 x c2d-standard-32 GCP Machine (8vCPU , 32GB Memory) and have made multiple attempts to tune the parameters for faster BigQuery to Clickhouse loads but even after increasing config Variables there seems to be no change in the system.

JOB_MAIN_CONTAINER_MEMORY_REQUEST=12g
JOB_MAIN_CONTAINER_MEMORY_LIMIT= 12g
JOB_MAIN_CONTAINER_CPU_REQUEST= 0.75
JOB_MAIN_CONTAINER_CPU_LIMIT= 0.80

MAX_SYNC_WORKERS=20
MAX_SPEC_WORKERS=20
MAX_CHECK_WORKERS=20
MAX_DISCOVER_WORKERS=20
MAX_NOTIFY_WORKERS=20

Similar Configs for Normalisation Container Variables also. We have run multiple syncs of multiple sizes but haven’t been able to increase the ETA of sync and is constant. Usage of the machine from airbyte to isn’t more than 40% of CPU and 4GBs of RAM (out of 32GB). With current test sync speeds Airbyte would take days to sync data for our needs (BigQuery to Clickhouse) 1) Anyway to configure the system in the best possible way for full utilization of the machine ? 2) What are the best practices for scaling Airbyte docker deployment to its full potential?

any insights on this folks ? We could use some tips here 🙂

Hello Shreshth Arora, it's been a while without an update from us. Are you still having problems or did you find a solution?