Based on the available information, Airbyte’s memory usage during syncs is primarily driven by the source worker, which can hold up to 10,000 records in memory at a time. For tables with large rows (e.g., JSON/BLOB columns), this can translate into several GB of RAM usage and can cause OOM or very high memory consumption during syncs.[Scaling guide] A few relevant points from the knowledge sources: 1. Source-side batching behavior ◦ Sources read data in batches; the exact batch size in bytes is not known ahead of time and depends on row size.[Memory request explanation] ◦ Destinations typically process batches of up to 10k records, which is where the memory spike often comes from for large rows.[Scaling guide] 2. Postgres connector fetch size / memory ◦ Older Postgres connectors had a fixed

fetch_size

of 1000 rows; newer versions use a dynamic fetch size that tries to use a portion of the JVM heap, which can lead to high and sometimes unpredictable memory usage.[Optimized Postgres performance; Low memory footprint; [PG fetchSize issue] ◦ There is an open issue noting that the Postgres source may allocate up to 60% of the JVM heap for its buffer, which can easily reach tens of GB if the pod/container heap is large.[PG fetchSize issue] 3. Connector resource limits ◦ You can control the memory available to each sync job via: ▪︎

JOB_MAIN_CONTAINER_MEMORY_REQUEST

▪︎

JOB_MAIN_CONTAINER_MEMORY_LIMIT

◦ These use Docker-style units, e.g.

JOB_MAIN_CONTAINER_MEMORY_REQUEST=28g

.[Low memory footprint] ◦ If you set a hard limit and the connector exceeds it, the job will be killed (you’ll see OOM / “Killed” in logs), as another user observed when setting

JOB_MAIN_CONTAINER_MEMORY_LIMIT=3g

.[Default 2GB limit] Given your symptoms (≈30 GB RAM used to sync a ~1.6 GB Postgres table, per your screenshot), this aligns with the known behavior where the Postgres source over-allocates memory for its fetch buffer.[PG fetchSize issue] What you can do Because the knowledge sources don’t describe a user-facing knob to directly tune the Postgres

fetch_size

, the options mentioned are mostly operational: 1. Constrain connector memory ◦ Set

JOB_MAIN_CONTAINER_MEMORY_LIMIT

and

JOB_MAIN_CONTAINER_MEMORY_REQUEST

to a reasonable value (e.g. somewhat below your VM RAM) so a single sync cannot consume the entire host. Use Docker-style units like

8g

,

16g

, etc.[Low memory footprint; Scaling guide] ◦ Be aware: if the table truly needs more than that (because of very large rows), the job may fail with OOM rather than complete.[Default 2GB limit] 2. Split workload into multiple connections ◦ One forum answer suggests creating multiple connections for different tables to keep per-job memory lower and parallelize at the connector level.[Progressively push data] ◦ This doesn’t reduce memory for a single problematic table, but it can help avoid having many large tables in one job. 3. Upgrade to latest Postgres connector / Airbyte ◦ Several posts mention improvements to the Postgres connector’s dynamic fetch size over time and bug fixes for memory issues.[Optimized Postgres performance; CockroachDB OOM] ◦ The open GitHub issue proposes reducing the buffer to 10% of heap with a 500MB cap, but the knowledge sources do not confirm whether this change has been released yet.[PG fetchSize issue] ◦ I’m therefore not certain which exact behavior your current version has; checking and upgrading to the latest stable Airbyte and

source-postgres

is recommended. 4. Be mindful of very large rows ◦ The docs note that tables with large row sizes (e.g. 0.5 MB per row) can require several GB of RAM for the 10k-record buffer.[Scaling guide] ◦ If your table has large JSON/BLOB columns, consider: ▪︎ Excluding those columns from the sync, or ▪︎ Moving them to a separate table/connection, if possible (not explicitly documented, but consistent with the guidance to split workloads).[Progressively push data] Because the knowledge sources don’t document a direct way to “stream” a single table in smaller, user-configurable batches from Postgres to BigQuery, I can’t provide a precise tuning parameter beyond the global job memory settings and version upgrades. If you can share your Airbyte version and

source-postgres

connector version, you can compare them against the versions mentioned in the GitHub issue to see if you’re on an affected release, and then try: • Lowering

JOB_MAIN_CONTAINER_MEMORY_LIMIT

to something like

16g

to prevent full-host exhaustion, and • Upgrading Airbyte / the Postgres connector to pick up the latest memory-behavior fixes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.