@kapa.ai I am running into OOM with the Airbyte Databricks lakehouse sink. Is there a way I can configure an increased heap size?

Yes, you can configure the memory settings for your Airbyte jobs by setting the

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables. These variables allow you to specify the memory request and limit for the main job container, which includes the source and destination connectors. Here's how you can do it: 1. Open your

.env

file which is located in the root directory of your Airbyte project. 2. Add or modify the following lines with your desired memory settings:

bash JOB_MAIN_CONTAINER_MEMORY_REQUEST=4g JOB_MAIN_CONTAINER_MEMORY_LIMIT=6g

In this example, the memory request is set to 4GB and the memory limit is set to 6GB. You can adjust these values based on your specific needs and the resources available on your machine. Please note that these settings will apply to all jobs run by Airbyte. If you want to configure memory settings for a specific connection, you can do so by following the guide on [Configuring Connector Resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources). Also, keep in mind that Java connectors, like the Databricks connector, follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. So, you might need to over-specify memory resources for these connectors. For more information, you can refer to the [Scaling Airbyte](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory) guide in the Airbyte documentation. Sources: - [Scaling Airbyte](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory) - [Github Issue #30814](https://github.com/airbytehq/airbyte/issues/30814) - [Github Issue #34874](https://github.com/airbytehq/airbyte/issues/34874) - [Github Issue #39128](https://github.com/airbytehq/airbyte/issues/39128) - [Forum Post: CockroachDB Source Connector Failures at >10M records](https://discuss.airbyte.io/t/1261) - [Github Discussion #34483](https://github.com/airbytehq/airbyte/discussions/34483) - [Forum Post: Source BigQuery - Temporary file size reached at sync](https://discuss.airbyte.io/t/3126) - [Github Issue #16652](https://github.com/airbytehq/airbyte/issues/16652) - [Forum Post: After Upgrading Airbyte to v0.34.4-alpha get error](https://discuss.airbyte.io/t/1107)