@kapa.ai The S3 source keeps getting OOMed killed

Based on the information provided in the knowledge sources, it appears that out-of-memory (OOM) issues are a known problem with Airbyte, including when using S3 as a source or destination. However, the knowledge sources don't specifically mention S3 source getting OOM killed. Let me provide some general insights about OOM issues in Airbyte that might be relevant to your situation: 1. Memory limits: Airbyte has default memory limits for containers, which can be adjusted. The default limit is often 2GB, but this may not be sufficient for large data volumes or complex operations. 2. Increasing memory: You can try increasing the memory limit for the source container. This can be done by setting environment variables such as

JOB_MAIN_CONTAINER_MEMORY_LIMIT

and

JOB_MAIN_CONTAINER_MEMORY_REQUEST

. 3. Debugging: You can check the job logs to confirm if the right memory configuration is being applied. The logs should show the resource requirements for the containers. 4. Gradual memory increase: Some users have reported gradual increases in memory usage over time, which might lead to OOM issues. 5. Version-specific issues: Some memory-related problems have been addressed in newer versions of Airbyte. Upgrading to the latest version might help resolve the issue. If you're still experiencing OOM issues after trying these steps, you may want to consider: - Breaking down your sync into smaller chunks of data - Investigating if there are any specific characteristics of your S3 data that might be causing high memory usage - Reaching out to Airbyte support or community forums with specific details about your setup and the exact error messages you're seeing For more detailed information on configuring resources and debugging connection issues, you can refer to the [Airbyte documentation on configuring connector resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources#debugging-connection-resources).