<#C01AHCD885S|ask-ai> When using Source Airbyte Co...
# ask-ai
m
#ask-ai When using Source Airbyte Connector as MongoDB & Destination Airbyte Connector as S3, does data syncing feature is available?
k
Yes, data syncing is available when using MongoDB as the source connector and S3 as the destination connector in Airbyte. Both connectors support the necessary sync modes for data transfer: - The MongoDB source connector supports both Full Refresh and Incremental sync modes, utilizing change data capture (CDC) for incremental updates and checkpointing for reliable full refreshes [MongoDB source replication methods](https://docs.airbyte.com/integrations/sources/mongodb-v2#replication-methods). - The S3 destination connector supports Full Refresh (Overwrite) and Incremental (Append) sync modes [S3 destination supported sync modes](https://docs.airbyte.com/integrations/destinations/s3#supported-sync-modes). This means you can set up a connection in Airbyte to sync data from MongoDB to S3 using either Full Refresh or Incremental sync modes, depending on your requirements. However, note that Incremental Append + Deduped mode is not supported by the S3 destination. If you need to move files and records together (for example, if your MongoDB contains unstructured attachments), Airbyte also supports syncing files and records together for certain connectors, but MongoDB is not listed among the sources that support file transfers [Sync files and records together](https://docs.airbyte.com/platform/using-airbyte/sync-files-and-records). In summary, standard data syncing between MongoDB and S3 is supported in Airbyte.
m
@kapa.ai What do you mean by Deduped mode?
k
A message from kapa.ai
In Airbyte, Deduped mode refers to a sync mode where the data in the final destination table is made unique per primary key. This means that, after syncing, each primary key will only have one row in the destination table. Deduplication is achieved by sorting the data using the cursor field (such as a timestamp or an updated_at column) and keeping only the latest version of each record. For example, in Full Refresh Overwrite + Deduped mode, all data in the destination table is replaced with new data from the source, and any duplicate records (based on the primary key) are removed, keeping only the most recent one. Similarly, in Incremental Append + Deduped mode, only new or updated records are synced, and the destination table is deduplicated so that each primary key appears only once, with the latest data. This deduplication process ensures that the destination table reflects the most up-to-date state of the source, without duplicate records for the same primary key. However, deduplication can incur additional costs in your data warehouse due to the extra processing required. > "The deduped variant also means that the data in the final table will be unique per primary key... This is determined by sorting the data using the cursor field and keeping only the latest de-duplicated data row." [Full Refresh - Overwrite + Deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite-deduped) > "Deduped means that data in the final table will be unique per primary key... This is determined by sorting the data using the cursor field and keeping only the latest de-duplicated data row." [Incremental Sync - Append + Deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)
m
@kapa.ai So if I change my Destination from S3 to ClickHouse does it means that it will support Incremental Sync - Append + Deduped?
k
A message from kapa.ai
Yes, if you change your destination from S3 to ClickHouse (using the new ClickHouse v2 connector), you will be able to use the Incremental Sync - Append + Deduped mode. The ClickHouse v2 destination connector fully supports all sync modes, including Incremental - Append + Deduped, and leverages ClickHouse’s native
ReplacingMergeTree
for efficient deduplication. This is a significant improvement over the legacy ClickHouse connector, which did not support this mode. > “All sync modes supported: Full Refresh (Overwrite and Append) and Incremental (Append and Append + Deduped) sync modes are now fully supported.” > “Incremental - Append + Deduped Yes Leverages `ReplacingMergeTree`” [ClickHouse destination documentation](https://docs.airbyte.com/integrations/destinations/clickhouse) So, switching your destination to ClickHouse v2 will enable you to use Incremental Sync - Append + Deduped mode for your data syncs.
m
@kapa.ai Can I alway extract data monthly based? I have a field in my database that stores the datetime, using that can I extract?
k
A message from kapa.ai
Yes, you can extract data on a monthly basis using Airbyte if your database has a datetime field. This is possible by configuring the connector with a
DatetimeBasedCursor
in the YAML configuration. You can specify the
start_datetime
,
end_datetime
, and set the
step
parameter to
"P1M"
(ISO 8601 duration for one month) or to daily if you want to partition by day. The sync will then extract data in time windows based on your configuration. For example, your configuration might look like:
Copy code
yaml incremental_sync:     type: DatetimeBasedCursor     start_datetime: "2022-01-01T00:00:00"     end_datetime: "2022-12-31T23:59:59"     datetime_format: "%Y-%m-%dT%H:%M:%S"     cursor_granularity: "PT1S"     step: "P1M"  # Monthly windows
This setup will extract data in monthly partitions based on your datetime field. The
cursor_field
should be set to the name of your datetime column. Airbyte will use this field to filter and partition the data accordingly during incremental syncs. You can also use request options to map the datetime field to API parameters if you are extracting from an API source [DatetimeBasedCursor documentation](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/incremental-syncs#datetimebasedcursor) [Filtering according to Cursor Field](https://docs.airbyte.com/platform/next/connector-development/config-based/understanding-the-yaml-file/incremental-syncs#filtering-according-to-cursor-field). So, as long as your source supports filtering or partitioning by a datetime field, you can extract data monthly using Airbyte’s incremental sync features.
m
@kapa.ai How should I decide which ec2 instance or aws service is suitable for my data transfer activity to host my self hosted airbyte?
k
A message from kapa.ai
To decide which EC2 instance or AWS service is suitable for hosting your self-hosted Airbyte deployment, consider the following recommendations from the official Airbyte documentation and community discussions: ### 1. Minimum Instance Requirements - Do not use small instances like t3.micro: These are not recommended and will likely result in poor performance and system freezes, even for basic testing. Users have reported issues with t3.micro, and Airbyte documentation suggests starting with larger instances. - Recommended instance types: - For testing: t2.medium (2 vCPUs, 4 GB RAM) is likely sufficient. - For production: t2.large (2 vCPUs, 8 GB RAM) or larger is recommended. For more demanding workloads, consider memory-optimized instances such as M7i or M7g types, especially if you expect to run multiple concurrent syncs or handle large data volumes [Forum: EC2 instance recommendations](https://discuss.airbyte.io/t/2334) [Enterprise setup prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites). ### 2. Resource Guidelines - CPU and Memory: Airbyte recommends running on instances with at least 2 cores and 8 GB of RAM for production. For best performance, 4 or more CPUs and at least 8 GB of memory are suggested [OSS Quickstart](https://docs.airbyte.com/platform/1.6/using-airbyte/getting-started/oss-quickstart). - Disk Space: Allocate a minimum of 30 GB of disk space per node, as Airbyte connector images and logs can consume significant storage, especially with multiple connectors [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). ### 3. Kubernetes vs. EC2 - Kubernetes: For production-ready and scalable deployments, Airbyte recommends using Amazon EKS (Elastic Kubernetes Service) with EC2 instances. This allows for better scaling, high availability, and easier management of resources. A typical production deployment uses at least 6 nodes, each with at least 2 cores and 8 GB RAM, across 2 or more availability zones [Enterprise setup prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites). - Direct EC2: For smaller or test deployments, you can run Airbyte directly on EC2 instances using Docker Compose, but ensure you meet the minimum resource requirements above. ### 4. Scaling Considerations - The number of concurrent syncs, the size of your data, and the types of connectors you use will impact your resource needs. Monitor CPU, memory, and disk usage, and scale up your instance size or node count as needed [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). ### 5. Summary Table | Use Case | Recommended Instance Type | Notes | |------------------|--------------------------|---------------------------------------------| | Testing | t2.medium | Not for production workloads | | Production | t2.large or larger | At least 2 vCPUs, 8 GB RAM, 30 GB disk | | High Performance | M7i / M7g (memory-optimized) | For large or concurrent sync workloads | | Kubernetes | EKS with 6+ nodes | Each node: 2+ vCPUs, 8+ GB RAM, 30+ GB disk | In summary: Start with at least a t2.large instance for production, and consider memory-optimized instances or a Kubernetes cluster (EKS) for scalability and reliability. Monitor your resource usage and scale up as your data transfer needs grow. If you have specific workload requirements (e.g., high concurrency, large datasets), opt for larger or more specialized instances as recommended in the documentation.
m
@kapa.ai Let's say my dataset is more than 5TB & I don't know how we can set high concurrency syncing on airbyte - so guid me for that too.
k
A message from kapa.ai
If your dataset is more than 5TB and you want to set up high concurrency syncing in Airbyte, here’s how you should approach it based on the official documentation and community guidance: --- ### 1. Infrastructure Sizing and Instance Selection - Cluster Size: For large-scale, production-ready deployments, Airbyte recommends using a Kubernetes cluster (e.g., Amazon EKS) with at least 6 nodes, each with a minimum of 2 cores and 8GB RAM. For high concurrency and large datasets, use memory-optimized instances (e.g., M7i/M7g) and consider scaling up both the number and size of nodes as needed. - Disk Space: Allocate at least 30GB of disk space per node, but for a 5TB+ dataset, you may need significantly more, depending on your sync patterns and connector images [Enterprise setup prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites) [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). --- ### 2. Configuring High Concurrency Syncing #### Key Environment Variables and Helm Values - MAX_SYNC_WORKERS: Controls the maximum number of concurrent sync jobs a worker can run. - worker replicaCount: Number of worker pods running in your Airbyte instance. - TEMPORAL_WORKER_PORTS: Should be set to at least the sum of all MAX_*_WORKERS across all workers. Example configuration (in
values.yaml
for Helm):
Copy code
yaml worker:   replicaCount: 10         # Number of worker pods   maxSyncWorkers: 400      # Max concurrent syncs per worker   maxCheckWorkers: 100   maxSpecWorkers: 100   maxDiscoverWorkers: 100   extraEnv:     - name: MAX_SYNC_WORKERS       value: "400"     - name: MAX_CHECK_WORKERS       value: "100"
- Resource Requests/Limits: Set appropriate CPU and memory requests/limits for your worker and job pods to ensure the cluster can handle the load and pods are scheduled efficiently. Example:
Copy code
yaml jobs:   resources:     limits:       cpu: 2500m       memory: 8Gi     requests:       cpu: 1000m       memory: 4Gi
- Kubernetes Pod Scheduling: Ensure your cluster can schedule up to
2 x <number-of-possible-concurrent-connections>
pods at once, as each sync job typically launches multiple pods [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). --- ### 3. Monitoring and Tuning - Monitor: Use infrastructure monitoring tools to track CPU, memory, and disk usage. If you see high resource usage or jobs getting stuck, scale up your nodes or adjust resource limits. - Iterate: Start with conservative concurrency settings and gradually increase them while monitoring system stability and throughput. --- ### 4. Additional Notes - Database and Object Storage: Use dedicated, scalable services like Amazon RDS for the Airbyte database and S3 for object storage. - Secrets Management: Use AWS Secrets Manager for storing connector secrets securely. --- ### 5. Summary Table | Parameter | Recommendation (for high concurrency) | |--------------------------|----------------------------------------------| | worker replicaCount | 10+ (scale as needed) | | MAX_SYNC_WORKERS | 100–400+ per worker (tune as needed) | | TEMPORAL_WORKER_PORTS | ≥ sum of all MAX_*_WORKERS | | Node Type | Memory-optimized (e.g., M7i/M7g) | | Node Count | 6+ (scale as needed) | | Disk Space per Node | 30GB minimum, more for large datasets | | CPU/Memory per Pod | Tune based on connector requirements | --- In summary: For syncing a 5TB+ dataset with high concurrency, deploy Airbyte on a robust Kubernetes cluster with memory-optimized nodes, set high values for
MAX_SYNC_WORKERS
and
worker replicaCount
, and monitor your resources closely. Adjust your configuration iteratively to achieve the desired throughput and stability. For more details and examples, refer to the official scaling and enterprise setup documentation [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale) [Enterprise setup prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites) [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte).