I keep getting this error when trying to sync large table over 50 Million rows, all I did is changing the cpu and memory request to 4 cpu and 4g memory. I'm using airbyte with docker and the source database is oracle and destination is local postgres, postgres also running on docker.

Stack Trace: java.lang.RuntimeException: org.postgresql.util.PSQLException: ERROR: unexpected data beyond EOF in block 26789 of relation base/224213/233904
2741 Hint: This has been seen to occur with buggy kernels; consider updating your system.
2742 Where: COPY public_raw__stream_fdebts_primium_micro_daily, line 234
2743 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.lambda$insertRecordsInternal$0(PostgresSqlOperations.java:92)
2744 at io.airbyte.cdk.db.jdbc.DefaultJdbcDatabase.execute(DefaultJdbcDatabase.kt:30)
2745 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.insertRecordsInternal(PostgresSqlOperations.java:81)
2746 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.insertRecordsInternalV2(PostgresSqlOperations.java:53)
2747 at io.airbyte.cdk.integrations.destination.jdbc.JdbcSqlOperations.insertRecords(JdbcSqlOperations.kt:231)
2748 at io.airbyte.cdk.integrations.destination.jdbc.JdbcBufferedConsumerFactory.recordWriterFunction$lambda$5(JdbcBufferedConsumerFactory.kt:302)
2749 at io.airbyte.cdk.integrations.destination.jdbc.JdbcInsertFlushFunction.flush(JdbcInsertFlushFunction.kt:19)
2750 at io.airbyte.cdk.integrations.destination.async.FlushWorkers.flush$lambda$6(FlushWorkers.kt:172)
2751 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
2752 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
2753 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
2754 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
2755 at java.base/java.lang.Thread.run(Thread.java:1583)
2756 Caused by: org.postgresql.util.PSQLException: ERROR: unexpected data beyond EOF in block 26789 of relation base/224213/233904
2757 Hint: This has been seen to occur with buggy kernels; consider updating your system.
2758 Where: COPY public_raw__stream_fdebts_primium_micro_daily, line 234
2759 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2713)
2760 at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:1299)
2761 at org.postgresql.core.v3.QueryExecutorImpl.endCopy(QueryExecutorImpl.java:1095)
2762 at org.postgresql.core.v3.CopyInImpl.endCopy(CopyInImpl.java:49)
2763 at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:185)
2764 at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:160)
2765 at io.airbyte.integrations.destination.postgres.PostgresSqlOperations.lambda$insertRecordsInternal$0(PostgresSqlOperations.java:90)

It looks like you're encountering an issue with syncing a large table from Oracle to a local PostgreSQL instance using Airbyte. The error message indicates a problem with the PostgreSQL database, specifically an "unexpected data beyond EOF" error, which can be related to buggy kernels. Here are some steps you can take to troubleshoot and potentially resolve this issue: 1. Update Your System: The error message suggests that this issue can occur with buggy kernels. Ensure that your system, including the kernel, is up to date. This can help prevent such low-level issues. 2. Check PostgreSQL Configuration: Ensure that your PostgreSQL instance is configured correctly to handle large data loads. You might need to adjust settings like

shared_buffers

,

work_mem

, and

maintenance_work_mem

to better handle large data operations. 3. Increase Memory Allocation: According to the [Airbyte documentation](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory), Java connectors in Airbyte will only use up to 1/4 of the host's allocated memory by default. You can customize this by setting the

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables to higher values. Ensure that your Docker containers have enough memory allocated. 4. Review Sync Logs: Detailed logs can provide more insights into what might be going wrong. Ensure that you have logging configured correctly and review the logs for any additional clues. 5. Consider Table Indexing: As mentioned in a [forum post](https://discuss.airbyte.io/t/551), creating an index on the cursor field in the source can speed up sync times and result in fewer failures. This might be particularly useful if your syncs are timing out or failing due to large data volumes. 6. Retry with Smaller Batches: If possible, try syncing the data in smaller batches. This can help isolate the issue and make it easier to manage large data volumes. 7. Check for Known Issues: Review the [Airbyte GitHub issues](https://github.com/airbytehq/airbyte/issues) for any similar problems reported by other users. There might be specific fixes or workarounds available. If the problem persists, consider reaching out to the Airbyte community or support for further assistance. You can open a new topic on the [Airbyte forum](https://discuss.airbyte.io/) if you need more help. By following these steps, you should be able to identify and resolve the issue with syncing large tables in Airbyte.