I am getting this error, when I load a large table from Oracle to DataBricks : 025-12-11 120714 destination INFO sync-operations-2 i.a.i.d.d.o.DatabricksStorageOperation(cleanupStage):158 Deleting Staging directory /Volumes/raw_data/airbyte_internal/coast_raw__stream_mad_staging/2025_12_11_10_8a825abf-3787-4e2c-ba96-a071dc78ac0c 2025-12-11 120714 destination ERROR main i.a.c.i.u.ConnectorExceptionUtil(logAllAndThrowFirst):37 Following exceptions occurred while finalizing the syncjava.util.concurrent.CompletionException: com.databricks.sdk.core.error.platform.BadRequest: Response from server (Bad Request) { "error_code" : "BAD_REQUEST", "message" : "The directory is not empty. This operation is not supported on non-empty directories.", "details" : [ { "@type" : "type.googleapis.com/google.rpc.ErrorInfo", "reason" : "FILES_API_DIRECTORY_IS_NOT_EMPTY", "domain":"**********" } ] }: Cannot construct instance of

com.databricks.sdk.core.error.ErrorDetail

, problem: Cannot invoke "java.util.Map.getClass()" because "m" is null at [Source: REDACTED (

StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION

disabled); line: 8, column: 3] (through reference chain: com.databricks.sdk.core.error.ApiErrorBody["details"]->java.util.ArrayList[0]) at io.airbyte.commons.concurrency.CompletableFutures$allOf$1.invoke(CompletableFutures.kt:46) at io.airbyte.commons.concurrency.CompletableFutures$allOf$1.invoke(CompletableFutures.kt:31) at io.airbyte.commons.concurrency.CompletableFutures.allOf$lambda$1(CompletableFutures.kt:31) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: com.databricks.sdk.core.error.platform.BadRequest: Response from server (Bad Request) { "error_code" : "BAD_REQUEST", "message" : "The directory is not empty. This operation is not supported on non-empty directories.", "details" : [ { "@type" : "type.googleapis.com/google.rpc.ErrorInfo", "reason" : "FILES_API_DIRECTORY_IS_NOT_EMPTY", "domain":"**********" } ] }: Cannot construct instance of

com.databricks.sdk.core.error.ErrorDetail

, problem: Cannot invoke "java.util.Map.getClass()" because "m" is null at [Source: REDACTED (

StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION

disabled); line: 8, column: 3] (through reference chain: com.databricks.sdk.core.error.ApiErrorBody["details"]->java.util.ArrayList[0]) at com.databricks.sdk.core.error.AbstractErrorMapper.apply(AbstractErrorMapper.java:43) at com.databricks.sdk.core.error.ApiErrors.getDatabricksError(ApiErrors.java:29) at com.databricks.sdk.core.ApiClient.executeInner(ApiClient.java:276) at com.databricks.sdk.core.ApiClient.getResponse(ApiClient.java:235) at com.databricks.sdk.core.ApiClient.execute(ApiClient.java:227) at com.databricks.sdk.core.ApiClient.DELETE(ApiClient.java:188) at com.databricks.sdk.service.files.FilesImpl.deleteDirectory(FilesImpl.java:45) at com.databricks.sdk.service.files.FilesAPI.deleteDirectory(FilesAPI.java:85) at com.databricks.sdk.service.files.FilesAPI.deleteDirectory(FilesAPI.java:73) at io.airbyte.integrations.destination.databricks.operation.DatabricksStorageOperation.cleanupStage(DatabricksStorageOperation.kt:159) at io.airbyte.integrations.base.destination.operation.AbstractStreamOperation.finalizeTable(AbstractStreamOperation.kt:342) at io.airbyte.integrations.base.destination.operation.DefaultSyncOperation.finalizeStreams$lambda$9$lambda$8(DefaultSyncOperation.kt:138) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)

This error is coming from Databricks when Airbyte tries to delete the staging directory at the end of the sync: >

The directory is not empty. This operation is not supported on non-empty directories.

From the available knowledge sources, there is no specific mention of this exact

FILES_API_DIRECTORY_IS_NOT_EMPTY

/

cleanupStage

failure pattern for the Databricks destination. The Databricks destination docs only state that Airbyte can optionally purge staging data, but do not describe known issues or workarounds for directory‑deletion failures. [[Databricks dest docs](https://docs.airbyte.com/integrations/destinations/databricks)] Given that: 1. Airbyte is calling

FilesAPI.deleteDirectory

during

cleanupStage

, and Databricks is rejecting deletion because it still sees files under that directory. 2. This happens when loading a large Oracle table, i.e., a lot of staging files are created. The knowledge sources do not provide: - A configuration flag to change cleanup behavior beyond the documented “Purge Staging Files and Tables” option. [[Databricks dest docs](https://docs.airbyte.com/integrations/destinations/databricks)] - A known bug report or guidance for this specific Databricks Files API error in Airbyte. Because of that, I can’t confidently give you an Airbyte‑specific fix from the docs. What you can safely try, based on the docs: - In the Databricks destination configuration, disable

Purge Staging Files and Tables

so Airbyte does not attempt to delete the staging directory at the end of the sync. [[Databricks dest docs](https://docs.airbyte.com/integrations/destinations/databricks)] - This may allow the sync to complete even if the directory is non‑empty, at the cost of leaving staging files/directories in your volume. You’d then manage cleanup on the Databricks side (e.g., lifecycle policies or periodic manual cleanup). If disabling purge does not resolve it, or if you require purge semantics, then the next step is to open an issue (or support ticket if you’re on Airbyte Cloud) including: - Full job logs. - Databricks workspace / volume / catalog path details (anonymized). - Confirmation of which Databricks runtime and Files API / volumes setup you’re using. As of the information in the knowledge sources, there isn’t a documented Airbyte configuration or version that explicitly fixes this

FILES_API_DIRECTORY_IS_NOT_EMPTY

error.