Replication using Airbyte OSS and Oracle destination 0.2.0 (for compliance with Oracle 19C) is very slow (K8S, POC env with Airbyte, dagster, dbt in WSL2 & Docker Desktop 32GB RAM)

Slow replication to an Oracle destination is a known challenge with Airbyte. Based on the knowledge sources, here are the relevant findings and suggestions: Known Performance Issues with Oracle Destination The Oracle destination connector has had several updates, and as of version 1.0.0 (April 2024), normalization was removed and the connector was updated to V2 Raw Table Format. [Oracle changelog] Make sure you're on the latest version. General Bottlenecks Observed A community member running a high-volume Oracle source sync (242 GB table) identified several bottlenecks: [Oracle sync performance] 1. Source-to-destination messaging overhead: Generating an

AirbyteMessage

object per row is slow — fetching 300,000 rows took ~30 seconds, but sending them took over 3 minutes. 2. Log verbosity: For every 1,000 rows, a log message is printed. For large tables (e.g., 300M rows), this generates 300k log messages, which can significantly impact performance. 3. Resource underutilization: Even with allocated CPUs and memory, containers were not fully utilizing the given resources (source used ~100% CPU but <400 MB RAM; destination used ~30% CPU). 4. Changing output format or destination (e.g., Local JSON vs S3) did not significantly improve throughput. Suggestions to Improve Performance 1. Tune Kubernetes Worker Parallelism If you're running on Kubernetes, you can increase job parallelism: [K8s parallelism] • Increase

MAX_SYNC_WORKERS

• Increase

SUBMITTER_NUM_THREADS

• Scale

airbyte-worker

replicas For example, to run 100 sync jobs:

Number of Workers: 3 SUBMITTER_NUM_THREADS = 200 MAX_SYNC_WORKERS = 100

2. Allocate More Resources to the Connection You can configure connection-specific CPU and memory requirements. One user applied this via SQL: [ingestion slowdown]

UPDATE connection  SET resource_requirements = '{"cpu_limit": "1", "cpu_request": "1", "memory_limit": "4Gi", "memory_request": "4Gi"}'  WHERE id = '<your-connection-id>';

3. Split Large Syncs Isolating each stream/table into a separate sync job can help with performance. [ingestion slowdown] 4. Check Oracle-Side Timeouts One reported error (

No more data to read from socket

) was caused by the Oracle database dropping the connection due to an exceeded timeout. Increasing Oracle's timeout configuration may help. [Oracle sync fails] 5. WSL2 + Docker Desktop Caveat The knowledge sources do not explicitly address WSL2/Docker Desktop-specific performance tuning for Airbyte. However, given that your POC environment is resource-constrained (shared 32GB RAM across Airbyte, Dagster, dbt, and Docker Desktop), resource contention is a likely contributing factor. Ensuring sufficient memory and CPU are allocated to Docker Desktop's WSL2 backend would be advisable. In summary, slow Oracle replication is a recognized issue in the community, and the primary levers are resource allocation, worker parallelism, and reducing sync scope per job. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.