What can I do to solve this issue: I've got two separate branches in Nessie, when writing data to S3 Data Lake. Each destination is setup to use it's own branch and namespace, however when the connections run - all catalogs are created in one branch -

main

. Is there a way to solve this so each destination writes to it's own branch?

This is a known bug in

destination-s3-data-lake

version 0.3.52 and earlier. The connector ignores the configured

main_branch_name

for Nessie catalogs and always commits to

main

. Root Cause In

S3DataLakeUtil.kt

, the branch was only set under the key

"nessie.ref"

. However, Apache Iceberg 1.11.0's

NessieCatalog.initialize()

strips the

"nessie."

prefix and reads the branch from the un-prefixed

"ref"

key. Since

"ref"

was never set, the Nessie client fell back to its default branch (

main

). [GitHub Issue] Solution This bug has been fixed in version 0.3.53 via PR #82733. The fix supplies the configured branch under both

"ref"

and `"nessie.ref"`:

NESSIE_REF to config.icebergCatalogConfiguration.mainBranchName,          // "ref" — what NessieCatalog reads
NessieConfigConstants.CONF_NESSIE_REF to config.icebergCatalogConfiguration.mainBranchName, // "nessie.ref"

Upgrading to version 0.3.53 of

destination-s3-data-lake

will resolve your issue. [PR Fix] Temporary Workaround (if you can't upgrade yet) If you need an immediate fix before upgrading, you can patch the connector locally by changing the property key in

S3DataLakeUtil.kt

to use the literal string `"ref"`:

"ref" to config.icebergCatalogConfiguration.mainBranchName

This is the same local patch that was validated by the community reporter — after deploying the patched image, syncs correctly committed to the configured branch instead of

main

. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.