Mehul Sanghvi
11/25/2025, 10:18 AMkapa.ai
11/25/2025, 10:18 AMMehul Sanghvi
11/25/2025, 10:20 AMkapa.ai
11/25/2025, 10:20 AMkapa.ai
11/25/2025, 10:21 AMMehul Sanghvi
11/25/2025, 10:22 AMkapa.ai
11/25/2025, 10:22 AMkapa.ai
11/25/2025, 10:22 AMReplacingMergeTree for efficient deduplication. This is a significant improvement over the legacy ClickHouse connector, which did not support this mode. > “All sync modes supported: Full Refresh (Overwrite and Append) and Incremental (Append and Append + Deduped) sync modes are now fully supported.” > “Incremental - Append + Deduped Yes Leverages `ReplacingMergeTree`” [ClickHouse destination documentation](https://docs.airbyte.com/integrations/destinations/clickhouse) So, switching your destination to ClickHouse v2 will enable you to use Incremental Sync - Append + Deduped mode for your data syncs.Mehul Sanghvi
11/25/2025, 10:25 AMkapa.ai
11/25/2025, 10:25 AMkapa.ai
11/25/2025, 10:26 AMDatetimeBasedCursor in the YAML configuration. You can specify the start_datetime, end_datetime, and set the step parameter to "P1M" (ISO 8601 duration for one month) or to daily if you want to partition by day. The sync will then extract data in time windows based on your configuration. For example, your configuration might look like: yaml incremental_sync: type: DatetimeBasedCursor start_datetime: "2022-01-01T00:00:00" end_datetime: "2022-12-31T23:59:59" datetime_format: "%Y-%m-%dT%H:%M:%S" cursor_granularity: "PT1S" step: "P1M" # Monthly windows This setup will extract data in monthly partitions based on your datetime field. The cursor_field should be set to the name of your datetime column. Airbyte will use this field to filter and partition the data accordingly during incremental syncs. You can also use request options to map the datetime field to API parameters if you are extracting from an API source [DatetimeBasedCursor documentation](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/incremental-syncs#datetimebasedcursor) [Filtering according to Cursor Field](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/incremental-syncs#filtering-according-to-cursor-field). So, as long as your source supports filtering or partitioning by a datetime field, you can extract data monthly using Airbyte’s incremental sync features.Mehul Sanghvi
11/25/2025, 10:28 AMkapa.ai
11/25/2025, 10:29 AMkapa.ai
11/25/2025, 10:29 AMMehul Sanghvi
11/25/2025, 10:31 AMkapa.ai
11/25/2025, 10:32 AMkapa.ai
11/25/2025, 10:32 AMvalues.yaml for Helm): yaml worker: replicaCount: 10 # Number of worker pods maxSyncWorkers: 400 # Max concurrent syncs per worker maxCheckWorkers: 100 maxSpecWorkers: 100 maxDiscoverWorkers: 100 extraEnv: - name: MAX_SYNC_WORKERS value: "400" - name: MAX_CHECK_WORKERS value: "100" - Resource Requests/Limits: Set appropriate CPU and memory requests/limits for your worker and job pods to ensure the cluster can handle the load and pods are scheduled efficiently. Example: yaml jobs: resources: limits: cpu: 2500m memory: 8Gi requests: cpu: 1000m memory: 4Gi - Kubernetes Pod Scheduling: Ensure your cluster can schedule up to 2 x <number-of-possible-concurrent-connections> pods at once, as each sync job typically launches multiple pods [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). --- ### 3. Monitoring and Tuning - Monitor: Use infrastructure monitoring tools to track CPU, memory, and disk usage. If you see high resource usage or jobs getting stuck, scale up your nodes or adjust resource limits. - Iterate: Start with conservative concurrency settings and gradually increase them while monitoring system stability and throughput. --- ### 4. Additional Notes - Database and Object Storage: Use dedicated, scalable services like Amazon RDS for the Airbyte database and S3 for object storage. - Secrets Management: Use AWS Secrets Manager for storing connector secrets securely. --- ### 5. Summary Table | Parameter | Recommendation (for high concurrency) | |--------------------------|----------------------------------------------| | worker replicaCount | 10+ (scale as needed) | | MAX_SYNC_WORKERS | 100–400+ per worker (tune as needed) | | TEMPORAL_WORKER_PORTS | ≥ sum of all MAX_*_WORKERS | | Node Type | Memory-optimized (e.g., M7i/M7g) | | Node Count | 6+ (scale as needed) | | Disk Space per Node | 30GB minimum, more for large datasets | | CPU/Memory per Pod | Tune based on connector requirements | --- In summary: For syncing a 5TB+ dataset with high concurrency, deploy Airbyte on a robust Kubernetes cluster with memory-optimized nodes, set high values for MAX_SYNC_WORKERS and worker replicaCount, and monitor your resources closely. Adjust your configuration iteratively to achieve the desired throughput and stability. For more details and examples, refer to the official scaling and enterprise setup documentation [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale) [Enterprise setup prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites) [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte).