Hi @kapa.ai how to build connector locally and push to my docker hub ?

Building a Connector Locally and Pushing to Docker Hub Here's how to build your connector locally and push it to Docker Hub: 1. Build the Connector Image You have a couple of options: Option A – Using

airbyte-cdk

(recommended for Python/manifest-only connectors):

cd airbyte-integrations/connectors/source-<your-connector>
airbyte-cdk image build

[Local Connector Dev] Option B – Using `airbyte-ci`:

airbyte-ci connectors --name source-<source-name> build

Once done, your image will be available locally as

airbyte/source-<source-name>:dev

. [Forum Post] Option C – Using

docker build

directly:

docker build . -t my-custom-connectors/source-custom:0.1.0

[Custom Connectors Guide] 2. Tag the Image for Docker Hub

docker tag my-custom-connectors/source-custom:0.1.0 <your-dockerhub-username>/source-custom:0.1.0

3. Authenticate to Docker Hub You can use Docker Desktop to sign in, or via CLI:

docker login

For automated/CI use, use a Docker Hub service account. [Custom Connectors Guide] If using

airbyte-ci

, you can provide credentials via environment variables:

export DOCKER_HUB_USERNAME=your_username
export DOCKER_HUB_PASSWORD=your_password

[Forum Post] 4. Push the Image

docker push <your-dockerhub-username>/source-custom:0.1.0

[Custom Connectors Guide] 5. Add the Connector to Airbyte UI Once pushed, go to Settings → Sources (or Destinations) → Add custom connector in the Airbyte UI, and provide: • Display name • Docker repository name (e.g.,

your-dockerhub-username/source-custom

) • Tag (e.g.,

0.1.0

) • Documentation URL [Forum Post] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

how to install airbyte-cdk ? @kapa.ai

Installing the airbyte-cdk CLI Step 1: Install uv First, install

uv

(required to install the

airbyte-cdk

CLI):

brew install uv

Step 2: Install the airbyte-cdk CLI Once

uv

is installed, run:

uv tool install --upgrade 'airbyte-cdk[dev]'

To see all available commands after installation:

airbyte-cdk --help

[CDK Install Docs] Using airbyte-cdk as a Python Dependency If you want to use the CDK as a dependency in your connector project (rather than as a CLI tool), you can add it via Poetry:

poetry init
poetry add airbyte-cdk

[Custom Connector Tutorial] > Note: If you encounter version errors, make sure you are using a compatible Python version (Python 3.9+). [Forum Post] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai how does airbyte's cdk build images ?

How Airbyte's CDK Builds Images The knowledge sources provide some details on this, though not exhaustively. Here's what is documented: Overview The

airbyte-cdk

image build method does not use a Dockerfile from the connector directory. Instead, it uses the Dockerfile templates stored in the docker-images/ directory of the Airbyte repo, combined with internal build logic. [CDK Image Build] How It Works 1. Base image declared in

metadata.yaml

– The connector declares its base image under `data.connectorBuildOptions.baseImage`connectorBuildOptions baseImage: docker.io/airbyte/python-connector-base:3.0.0@sha256:...Using the full digest (

sha256

) ensures build reproducibility — the same commit always uses the same base image. [Migration Guide] 2. No Dockerfile needed – Declaring the base image in

metadata.yaml

makes a connector-level

Dockerfile

obsolete. The CDK handles the build internally. [Migration Guide] 3. Multi-arch builds – The

airbyte-cdk image build

command automatically builds both

arm64

and

amd64

images, defaulting the

dev

tag to

arm64

(for Mac M-series laptops). [Docker Images README] 4. Optional build customization – If a connector needs specific system dependencies or environment variables, a

build_customization.py

module can be added. It supports two hooksExampleasync def pre_connector_install(base_image_container: Container) -> Container: return await base_image_container.with_env_variable("MY_VAR", "value") async def post_connector_install(connector_container: Container) -> Container: return await connector_container.with_env_variable("MY_POST_VAR", "value")[Migration Guide] ◦

pre_connector_install

– mutates the base image before connector install ◦

post_connector_install

– mutates the container after connector install Running the Build

cd airbyte-integrations/connectors/source-<your-connector>
airbyte-cdk image build

[Docker Images README] > Note: The knowledge sources do not go into deeper internal implementation details of the CDK build process beyond what is described above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai the airbyte-cdk image build command is stuck here Building Image for Connector: airbyte/source-microsoft-sharepoint:dev Downloading Dockerfile and .dockerignore templates from GitHub...

I'm sorry, but the knowledge sources do not contain enough information to provide a confident answer about why

airbyte-cdk image build

gets stuck at the "Downloading Dockerfile and .dockerignore templates from GitHub..." step specifically. However, based on the general context available, here are some things worth checking: Possible Causes & Things to Try 1. Network/connectivity issue – The CDK is trying to download Dockerfile templates from GitHub. If your network is slow, behind a firewall, or GitHub is rate-limiting your requests, this step could hang.curl -I https://raw.githubusercontent.com/airbytehq/airbyte/master/docker-images/Dockerfile.python-connector ◦ Try checking if you can reach GitHub: 1. Proxy or firewall restrictions – If you're behind a corporate proxy, the CDK may not be able to reach GitHub. Make sure your proxy settings are configured correctly. 2. Outdated

airbyte-cdk

version – Try upgrading to the latest version:uv tool install --upgrade 'airbyte-cdk[dev]' 3. Retry the command – Sometimes a transient network issue causes the hang. Cancel with

Ctrl+C

and retry. > For more targeted help, I'd recommend: > • Checking the Airbyte GitHub Issues for similar reports > • Asking in the Airbyte Slack community where the engineering team is active 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai its building an amd64 image, i want arm

The knowledge sources mention that the

airbyte-cdk image build

command automatically builds both

arm64

and

amd64

images, defaulting the

dev

tag to

arm64

on Mac M-series laptops. [Docker Images README] However, the knowledge sources do not explicitly document a flag or option to force

arm64

-only builds with the

airbyte-cdk image build

command. Things to Try 1. Use

docker buildx

directly to build for `arm64`:docker buildx build --platform linux/arm64 -t airbyte/source-microsoft-sharepoint:dev . 2. Check if there's an

--architecture

flag – the

airbyte-ci

tool has this flag:airbyte-ci connectors --name=source-microsoft-sharepoint build --architecture linux/arm64But it's unclear if

airbyte-cdk image build

supports the same flag. > I'd recommend checking the CLI help output for available options: >

airbyte-cdk image build --help

> Or reaching out on the Airbyte Slack community for more specific guidance, as the knowledge sources don't cover this in enough detail. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

This is how I build with architecture different than the default one

airbyte-ci connectors --name=source-foo build --architecture=linux/amd64