@kapa.ai How airbyte's connector works ? I ran a job which had to do a historical sync of data.. which was around 34GB... now will it load all 34GB in memory at once and then starts loading ?

How Airbyte Handles Large Data Syncs No, Airbyte does not load all 34GB into memory at once. Here's how it works: Source Side (Reading Data) Source connectors read data in batches, not all at once. For JDBC-based connectors (MySQL, Postgres, etc.), this is controlled by a

fetchSize

parameter. Airbyte uses a dynamic fetch size computation to determine how many records to pull per batch. [Scaling Airbyte] The key memory concern on the source side is that the Source worker reads up to ~10,000 records in memory at a time. So the memory required depends on your average row size, not the total dataset size. For example: > A table with an average row size of 0.5MB will require

0.5 × 10,000 / 1,000 = 5GB

of RAM. [Scaling Airbyte] Destination Side (Writing Data) Destination connectors also process records in batches — currently set at 10,000 records per batch. This means large datasets are written incrementally, not all at once. [Memory & Batching] What Can Go Wrong (OOM Errors) Even though data is batched, OOM (Out of Memory) errors can still occur if: • Your row sizes are very large (wide tables with big columns) • The connector doesn't support chunked reads (e.g., the ClickHouse connector has this known issue) [ClickHouse Issue] • Memory buffers accumulate across multiple tables in a single sync [MySQL Ingest] Recommendations for Your 34GB Sync 1. Increase VM/container memory — use

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

to allocate more memory to connector pods. At least 16GB RAM is recommended for large syncs. [MySQL Ingest] 2. Split connections — create multiple Airbyte connections for different subsets of tables to parallelize and reduce per-sync memory pressure. [Postgres Memory] 3. Use Incremental sync after the initial historical load — subsequent syncs will only process new/changed data, drastically reducing memory needs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what about databricks connector ? how does that work when it comes to load historical data ?

Databricks Destination Connector — Historical Data Loading The knowledge sources do not contain enough detailed information specifically about how the Databricks destination connector handles historical/large data loads internally. However, here is what can be gathered: What the Knowledge Sources Tell Us • The Databricks destination connector writes each stream to its own Delta table. [Databricks Docs] • Like other destination connectors, it processes records in batches of ~10,000 records at a time — it does not load all data into memory at once. [Memory & Batching] • One community report noted that syncing 300MB took ~140 seconds (~2.15 MB/s) and 2GB took ~1h 15min with the S3 → Databricks connector, suggesting throughput can be relatively slow for large datasets. [Sync Performance] Supported Sync Modes for Historical Load For a full historical sync, you would typically use: [Databricks Docs] Key Considerations for 34GB Historical Sync • Expect slower throughput (~2 MB/s based on community reports) — a 34GB sync could take many hours. [Sync Performance] • Memory pressure comes primarily from the source side, not the Databricks destination side. • Ensure you have sufficient memory allocated via

JOB_MAIN_CONTAINER_MEMORY_LIMIT

. [Configuring Resources] For more granular details on Databricks connector internals (e.g., staging behavior, batch flush thresholds), I'd recommend checking the Airbyte Databricks connector documentation directly or raising a question in the Airbyte community forum. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai Help me understand this... my Airbyte is running in k8s.. I have separate node-pool for running connectors pods.. so that they don't hit memory or core constratins... machine is 4 vCU and 16GB ram... I have 3 machines in this node-pool to take care of the workload... now help me understand whether this is sufficient to sync 35 GB of data from BQ table to databricsk lakehouse via Airbyte connection >?

Is Your Setup Sufficient for a 35GB BQ → Databricks Sync? The knowledge sources don't contain specific benchmarks for BigQuery → Databricks syncs, so I'll reason through this based on what is documented about Airbyte's resource usage on Kubernetes. Your Setup Summary Pod Requirements Per Sync Each concurrent sync in Airbyte on Kubernetes spins up 3 pods: •

orchestrator

pod •

read

(source) pod — reads from BigQuery •

write

(destination) pod — writes to Databricks [Enterprise Scaling] Memory Concern — The Critical Part The main memory concern is on the source (BigQuery read) pod. Airbyte reads up to 10,000 records at a time into memory. So the memory needed depends on your average row size, not the total 35GB directly: >

Memory needed = avg_row_size_MB × 10,000 / 1,000

For example: • Avg row = 0.1 MB → ~1 GB RAM needed • Avg row = 0.5 MB → ~5 GB RAM needed • Avg row = 1 MB → ~10 GB RAM needed [Scaling Airbyte] With 16 GB per node, you have reasonable headroom for most row sizes, but large row sizes could still cause OOM errors. Kubernetes Pod Scheduling Airbyte recommends your cluster can schedule up to: >

2 × <number of concurrent connections>

pods at once [Scaling Airbyte] For a single 35GB sync, you need only 3 pods — well within your 3-node cluster's capacity. Recommendations 1. Check your average row size — this is the most important factor. If rows are large (>0.5 MB), consider increasing memory limits:

global:
  jobs:
    resources:
      limits:
        cpu: "2"
        memory: "8Gi"
      requests:
        cpu: "500m"
        memory: "4Gi"

[Enterprise Scaling] 2. Monitor actual usage — check your job logs for the resource configuration being applied. It will look like:

[source] image: airbyte/source-bigquery:x.x.x resources: ResourceRequirements(limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.5})
[destination] image: airbyte/destination-databricks:x.x.x resources: ...

[Debugging Resources] 3. Expect a long sync time — 35GB historical syncs can take several hours depending on connector throughput. This is normal for an initial full refresh. 4. Plan for incremental syncs after — once the historical load is done, switch to incremental mode to avoid re-syncing 35GB every time. Bottom Line Your setup (3 × 4vCPU/16GB nodes) is likely sufficient for a single 35GB sync, provided your row sizes are not extremely large. The main risk is OOM on the source pod if rows are wide. I'd recommend tuning

JOB_MAIN_CONTAINER_MEMORY_LIMIT

based on your actual row size before running the sync. [Configuring Resources] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai how much orechestrator and init containers require CPU ? ~2vCPU ?

Orchestrator & Init Container CPU Requirements Based on the knowledge sources, here is what is documented: Orchestrator Container According to Airbyte's own benchmarks, the orchestrator is relatively lightweight compared to source/destination containers. In their faster sync speed benchmarks: Previously it was 4 CPU / 4 GB RAM, but Airbyte has significantly reduced orchestrator resource needs with recent optimizations. [Faster Sync Speed] Init Container From actual job logs captured in the knowledge sources, the init container typically runs with:

[initContainer] resources: limits={memory=2Gi, cpu=0.5}, requests={memory=2Gi, cpu=0.3}

[GitHub Issue] Known Issue ⚠️ There is a known open bug where

REPLICATION_ORCHESTRATOR_CPU_REQUEST

and

REPLICATION_ORCHESTRATOR_CPU_LIMIT

environment variables are not always applied correctly — the workload-launcher defaults to

cpu=2

for both request and limit regardless of configuration. [GitHub Issue] Bottom Line • Orchestrator: ~*1 vCPU* is sufficient (not 2) • Init container: ~*0.3–0.5 vCPU* • However, due to the known bug, you may see 2 vCPU being allocated by default even if you configure lower values 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai if there's a known issue:

Known Issue ⚠️
There is a known open bug where REPLICATION_ORCHESTRATOR_CPU_REQUEST and REPLICATION_ORCHESTRATOR_CPU_LIMIT environment variables are not always applied correctly — the workload-launcher defaults to cpu=2 for both request and limit regardless of configuration.

then I can't assign 2vCPU directly to jobs.. because theat will lead to resource exhaustion

Resource Exhaustion Risk — You're Right to Be Concerned Your concern is valid. Let's break down the actual CPU consumption per sync based on the known bug behavior: Default CPU Allocation Per Sync (Due to Bug) [GitHub Issue] Impact on Your 3-Node Cluster (12 vCPU total) With the bug causing 2 vCPU default for orchestrator, running even 2-3 concurrent syncs could exhaust your node pool's CPU, causing pods to stay in

Pending

state with

Insufficient CPU

— exactly as reported by other users. [GitHub Issue] Workarounds Available Since the Helm/env var configuration is unreliable due to the bug, the community-confirmed workarounds are: 1. Set resource limits directly in the database (most reliable)

UPDATE connection
SET resource_requirements = '{
  "cpu_request": "0.25",
  "cpu_limit": "0.9",
  "memory_request": "512Mi",
  "memory_limit": "1850Mi"
}'::jsonb
WHERE id = '<your-connection-id>';

[GitHub Issue] > ⚠️ Note: This applies to source/destination containers only, not the orchestrator. 2. Use a Kubernetes Mutating Webhook to intercept and patch replication pods before they start — this is the only confirmed way to control orchestrator container resources until the bug is fixed. [GitHub Issue] 3. Use whole CPU units instead of millicores when configuring via env vars, as millicore parsing is also broken:

workloadLauncher:
  extraEnv:
    - name: JOB_MAIN_CONTAINER_CPU_LIMIT
      value: "1"       # Use "1" NOT "1000m"
    - name: JOB_MAIN_CONTAINER_CPU_REQUEST
      value: "1"

[GitHub Issue] Bottom Line Given the bug, for your 4 vCPU / 16 GB nodes, I'd recommend: • Running only 1 sync at a time on each node until the bug is fixed • Applying the database-level workaround for source/destination containers • Monitoring pod scheduling closely for

Insufficient CPU

warnings 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements