cant build destination-s3 (1.9.7) locally, missing dependencies

The knowledge sources don’t contain step‑by‑step logs for your specific

destination-s3:1.9.7

build failure, but they do show closely related dependency/build issues and how they were handled. From similar cases, there are two common dependency problems when building destination connectors locally: 1. Python connectors with native deps (example: Authlib / cffi) In one forum thread, a Python destination failed to build because a transitive dependency (

cffi

) couldn’t compile: missing

ffi.h

in the Docker build image. The fix was to customize the connector Docker image to install the required system libraries (libffi headers, etc.) before

pip install

runs.[Cannot build python destination] If your

destination-s3

build is failing on a similar

ffi.h: No such file or directory

/

cffi

error, you’ll need to: ◦ Add the necessary OS packages in the base image used by the connector (e.g. via the Dockerfile used by the Gradle

airbyteDocker

task). ◦ Rebuild the base image, then rebuild the connector. 1. Java connectors needing base images built locally For Java destinations (like

destination-oracle

in the example), the build can fail because required base images such as

airbyte/integration-base-java:dev

are not present locally. The fix there was to: ◦ First build the Java connector (or its base) so Gradle’s

airbyteDocker

task can build the base image. ◦ Or explicitly run

docker build . -t airbyte/<connector-name>:dev

in the connector directory to force building the base images locally.[Unable to create dev image] For

destination-s3

specifically: • It is a Java/Kotlin connector that uses the Bulk CDK. The changelog shows 1.9.7 is a normal release (CDK 0.2.1) and doesn’t mention any special build steps or extra dependencies beyond the usual Java connector toolchain.[S3 changelog] • There is no explicit doc in the provided sources listing extra system packages you must install to build

destination-s3:1.9.7

. Given that, the most likely issues (based on the examples) are: 1. Missing base images / Gradle Docker task failure./gradlew airbyte integrationsconnectorsdestination s3buildThis should build any required base images (e.g.

integration-base-java

) via the

airbyteDocker

tasks, similar to the Oracle example.[Unable to create dev image]docker build . -t airbyte/destination-s3:devThen inspect the error near the end of the Docker build; in the Oracle thread this surfaced the real missing dependency.[Unable to create dev image] ◦ Make sure you run the Gradle build from the repo root: ◦ If that fails, go into

airbyte-integrations/connectors/destination-s3

and run: 1. Missing system libraries for a transitive dependency ◦ If your error log looks like the

cffi

/

ffi.h

example, you’ll need to extend the base image used by the connector to install the missing system packages before the dependency is built.[Cannot build python destination] Because the knowledge sources don’t include your exact error output, I can’t say precisely which dependency is missing. To get more targeted help: • Run either: ◦

./gradlew :airbyte-integrations:connectors:destination-s3:build

, or ◦

docker build . -t airbyte/destination-s3:dev

inside the connector folder, • Then share the full error block around the first failure (especially any “fatal error: … No such file or directory” or “could not find …” messages). With that log, it should be possible to map your issue directly to one of the known patterns above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai i had to modify build.gradle and add this

dependencies {
  // useLegacyTaskLoader excludes bulk-cdk-core-load, which on the published-CDK path
  // also drops its transitive bulk-cdk-core-base (io.airbyte.cdk.command.Configuration +
  // micronaut + kotlin-logging). Re-add it explicitly (the plugin does this for cdk=local).
  implementation "io.airbyte.bulk-cdk:bulk-cdk-core-base:${cdkVersion}"
}

but now test fails

You’re running into an area where the knowledge sources are incomplete for your exact local setup, so I’ll anchor on what is documented and call out where I’m inferring. From the docs and PRs: • Bulk CDK destinations are expected to get their CDK dependencies entirely from the

airbyte-bulk-connector

plugin +

cdkVersion

, not from manual

implementation

lines.[Bulk CDK; Dataflow impl ref] • For a typical DB destination, the

build.gradle.kts

is very small and does not declare CDK artifacts directly; the plugin wires them in based on

cdkVersion

and

toolkits

.[Getting started] • When

cdkVersion=local

, the plugin uses project references like `airbyte cdkbulkcoreload`; when pinned (e.g.

1.0.13

), it pulls the published artifacts like

bulk-cdk-core-load

and the relevant toolkits.[Dataflow impl ref; prove‑fix PR] Your manual addition:

dependencies {
  implementation "io.airbyte.bulk-cdk:bulk-cdk-core-base:${cdkVersion}"
}

is not shown anywhere in the official examples, and the docs explicitly say “No need to manually declare CDK dependencies – the plugin handles it.”[Getting started] That strongly suggests: 1. The root cause of your original “missing dependencies” was likely a mis‑pinned or mismatched

cdkVersion

/ plugin configuration, not the absence of

bulk-cdk-core-base

in

build.gradle

. 2. Adding

bulk-cdk-core-base

manually can easily create version or classpath conflicts with what the plugin is already bringing in, which would explain tests now failing. Unfortunately, the knowledge sources don’t include the exact

destination-s3

build.gradle.kts

for 1.9.7 or the test failure output, so I can’t say exactly which test is failing or why. But based on the docs, the recommended path is: 1. Remove the manual

bulk-cdk-core-base

dependency. Let the

airbyte-bulk-connector

plugin manage CDK artifacts. 2. Verify your

cdkVersion

and plugin wiring:./gradlew airbyte integrationsconnectorsdestination s3dependencies \ --configuration runtimeClasspath | grep bulk-cdkYou should see published artifacts likeio.airbyte.bulk cdkbulk-cdk-core-load:cdkVersionand not

project :airbyte-cdk:bulk:core:load

unless you intentionally set

cdkVersion=local

.[Dataflow impl ref] ◦ In

airbyte-integrations/connectors/destination-s3/gradle.properties

, ensure you have a valid pinned version (for 1.9.7 this was

0.2.1

, later bumped to

1.0.13

in 1.9.8).[CDK bump PR] ◦ From the repo root, run: 1. If you really need

cdkVersion=local

for development, don’t add

bulk-cdk-core-base

manually; instead: ◦ Set

cdkVersion=local

in

gradle.properties

. ◦ Let the plugin resolve

:airbyte-cdk:bulk:core:load

and the toolkits from the local projects.[Dataflow impl ref] 1. *Refresh Gradle deps after changing CDK wiring:*The Java CDK docs note that after switching between local and pinned CDK, you may need to refresh dependencies with:./gradlew airbyte integrationsconnectorsdestination s3build --refresh-dependencies[Java CDK dev] At this point, if tests still fail, the missing piece is the actual test error output. The docs do describe the different test contexts (component vs integration vs Docker) and what kinds of DI / classpath issues they catch, but not specific failures for your case.[Troubleshooting tests] To go further in a precise way, I’d need: • The exact

cdkVersion

you’re using. • The full

build.gradle(.kts)

for

destination-s3

. • The first failing test’s stack trace (especially any

ClassNotFoundException

,

NoSuchMethodError

, or Micronaut DI errors). Without those, the safest advice grounded in the docs is: revert the manual CDK dependency, rely on the plugin +

cdkVersion

, verify the resolved dependencies with

:dependencies

, and then debug any remaining test failure from the concrete error message. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai the failing tests

Sorry @guifesquet, I encountered an error processing the file(s) you uploaded: File type is not supported

@kapa.ai this is the kind of errors #9 [3/4] CS3V2WriteTestCsvRootLevelFlattening > testAppendSchemaEvolution() FAILED java.lang.ExceptionInInitializerError Caused by: java.nio.file.NoSuchFileException: secrets/s3_dest_iam_role_credentials_for_assume_role_auth.json S3V2WriteTestParquetUncompressed > testAppendSchemaEvolution() FAILED java.lang.NoClassDefFoundError: Could not initialize class io.airbyte.integrations.destination.s3_v2.S3V2TestUtils Caused by: java.lang.ExceptionInInitializerError: Exception java.nio.file.NoSuchFileException [in thread "ForkJoinPool-1-worker-12"]

Those failures are coming from the tests, not from the connector code itself: they’re trying to read a secrets file that doesn’t exist on your machine. From the error: •

NoSuchFileException: secrets/s3_dest_iam_role_credentials_for_assume_role_auth.json

•

S3V2TestUtils

fails to initialize because it can’t load that file. This matches the general pattern in the destination connector writer docs: integration/component tests expect a

secrets/

directory with a JSON config file, and that directory is git‑ignored so you must create it locally before running tests.[Database setup; Test config file] For

destination-s3

, the tests are looking specifically for:

airbyte-integrations/connectors/destination-s3/secrets/s3_dest_iam_role_credentials_for_assume_role_auth.json

To get past these errors: 1. In the

destination-s3

connector directory, create the

secrets

folder:cd airbyte-integrations/connectors/destination-s3 mkdir -p secrets 2. Create the file

secrets/s3_dest_iam_role_credentials_for_assume_role_auth.json

with valid test credentials for your environment. The exact schema isn’t shown in the provided sources, but it will be whatever

S3V2TestUtils

expects (typically similar to the connector’s config spec: bucket, region, and IAM role / credentials). The important part for the error you’re seeing is that the file exists and is readable. 3. Make sure

secrets/

is in

.gitignore

so you don’t commit credentials (this is how other destination connectors handle test secrets).[Test config file] 4. Re‑run the tests:./gradlew airbyte integrationsconnectorsdestination s3test If you still see failures after the

NoSuchFileException

is resolved, they will be due to the contents of that JSON (e.g., invalid or unauthorized AWS credentials), but the missing‑file error itself is fixed by creating the expected secrets file. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.