@kapa.ai, getting 504 gateway timeout error when the image is fetched from docker hub

The knowledge sources only give a small hint about 504 errors and timeouts; they don’t provide a full, step‑by‑step fix, so the guidance below is partially inferred. A 504 Gateway Timeout when Airbyte fetches a connector image usually means the proxy in front of Airbyte (Nginx/basic‑auth proxy) is timing out while the backend tries to pull the image. From the forum, one suggested fix is to increase the proxy timeout via the

BASIC_AUTH_PROXY_TIMEOUT

environment variable in your Airbyte deployment: [[Nginx 504 timeout](https://discuss.airbyte.io/t/3305)]

bash # in your .env (for Docker deployment) BASIC_AUTH_PROXY_TIMEOUT=600

Then restart your Airbyte stack so the new value is applied. If that doesn’t help, the knowledge sources don’t provide more specific troubleshooting steps for 504s when pulling from Docker Hub. In that case, you’d need to additionally check: - Network connectivity from the Airbyte host to Docker Hub - Any external reverse proxy/load balancer timeouts in front of Airbyte But those checks are outside what’s documented in the provided sources.

To change the timeout in Airbyte depends on how you deployed it. ### 1. Docker / Docker Compose (Airbyte proxy 504) For the built‑in Airbyte Nginx/basic‑auth proxy, you can increase the timeout with the

BASIC_AUTH_PROXY_TIMEOUT

environment variable in your

.env

file. [[Proxy 504 fix](https://discuss.airbyte.io/t/3305)] In your Airbyte directory: 1. Open

.env

and set, for example:

bash    BASIC_AUTH_PROXY_TIMEOUT=600

2. Restart Airbyte (e.g.

docker compose down && docker compose up -d

). This increases how long the proxy waits before returning a 504. ### 2. Kubernetes / Helm – schema discovery & API timeouts If your 504 is related to long‑running schema discovery or API calls, you can increase the server timeouts via Helm values: - Helm chart v1:

yaml   server:     extraEnv:       - name: HTTP_IDLE_TIMEOUT         value: 20m       - name: READ_TIMEOUT         value: 30m

- Helm chart v2:

yaml   server:     httpIdleTimeout: 20m     extraEnv:       - name: READ_TIMEOUT         value: 30m

[[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#schema-discovery-timeouts)] Apply the updated values with

helm upgrade

. If your 504 is specifically from an external ingress (e.g. nginx ingress controller) or an OAuth proxy, the knowledge sources only show examples of people extending those timeouts, but don’t give a definitive, Airbyte‑specific procedure beyond setting the above variables. [[Review schema 504](https://github.com/airbytehq/airbyte/issues/55690); [Gateway timeout](https://github.com/airbytehq/airbyte/issues/58627)]

The knowledge sources don’t state a specific “best” or “recommended” Airbyte platform tag to use when building a custom image for the new ClickHouse destination, so I can’t name an exact tag with high confidence. What is clear from the docs: - The new ClickHouse connector is the v2 connector (completely rewritten on the Bulk CDK) and is now GA and available in both Cloud and OSS. [[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse); [ClickHouse GA note](https://github.com/airbytehq/airbyte/discussions/62035)] - The old ClickHouse connector is archived and its last versions (e.g.

1.1.0

) are explicitly marked as deprecated. [[ClickHouse v1 archived](https://docs.airbyte.com/integrations/destinations/clickhouse-deprecated#changelog)] For building and testing connector changes locally, the platform docs describe a generic, version‑independent workflow: 1. Fork and clone the

airbyte

repo. 2. Build your connector image with `airbyte-ci`:

bash    airbyte-ci connectors --name destination-clickhouse build

This produces a local image like:

text    airbyte/destination-clickhouse:dev

[[Local dev 2.0](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] 3. Load it into your local

abctl

cluster and set the connector’s Docker image tag to

dev

(or another tag you choose) in Workspace settings → Destinations. [[Local dev 2.0](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] This flow does not depend on a particular Airbyte OSS tag in the docs; it just assumes you’re on a recent platform version that already includes the new ClickHouse destination. If you specifically need: - A stable connector version: use one of the released v2 ClickHouse destination versions from its changelog (e.g.

2.1.x

) as the base tag for your custom image, not the archived

1.x

line. [[ClickHouse v2 changelog](https://docs.airbyte.com/integrations/destinations/clickhouse#changelog)] - A matching Airbyte OSS version: the knowledge sources don’t map ClickHouse v2 versions to exact Airbyte OSS tags, so I can’t reliably tell you “use Airbyte OSS tag X with ClickHouse 2.Y.Z”. So, based on what’s documented: - Use a recent Airbyte OSS release that already ships the new ClickHouse destination. - Build your modified connector with

airbyte-ci

and tag it (e.g.

dev

or

2.1.18-custom

). - Configure that tag in Workspace settings → Destinations for ClickHouse. If you share which Airbyte OSS version you’re running now, I can at least confirm from the docs whether it’s new enough to have the v2 ClickHouse connector, but I still won’t be able to point to a single “officially recommended” OSS tag beyond that.

The knowledge sources you provided don’t contain any specific information about the error message: >

Failure in destination: Failed to initialize connector operation

So I can’t say exactly what is failing in your case. What I can share from the docs that is relevant to custom / modified destination connectors: 1. Make sure your custom image is actually built and available For Java destinations built from the Airbyte repo, you should build from the project root:

bash    ./gradlew :airbyte-integrations:connectors:destination-<name>:build

This compiles the code and builds the Docker image

airbyte/destination-<name>:dev

. [[Build Java dest 2.0](https://docs.airbyte.com/platform/2.0/connector-development/tutorials/building-a-java-destination#step-2-build-the-newly-generated-destination)] 2. If you run Airbyte via

abctl

(kind), load the image into the cluster If Airbyte is running in kind, you must load your local connector image into the cluster:

bash    kind load docker-image airbyte/destination-<name>:dev -n airbyte-abctl

[[Load into kind 2.0](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#loading-connector-docker-containers-into-kind)] If this step is skipped, the pod may fail to start or pull the wrong image, which can surface as initialization failures. 3. Point Airbyte to your custom image tag In local development, after building and loading the image, you must set the connector to use that tag: - Go to Workspace settings → Destinations. - Edit your destination connector. - Set the Docker image tag to

dev

(or whatever tag you used). - Save the change. [[Developing locally 2.0](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] If the UI still points to a different tag (e.g. the default from Docker Hub), Airbyte won’t use your local image. 4. Clear spec cache if you changed the connector spec If you modified

spec

(config schema) and Airbyte still uses the old one, you may see confusing initialization errors. To refresh: - Keep the version set to

dev

. - Click Change again to force a spec refresh. [[Developing locally 2.0](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-specification-caching)] --- Because the knowledge sources don’t include this exact error string, I can’t give a more precise root cause. To go further, I’d need: - The full error stack trace from the job logs. - Which destination (e.g. ClickHouse v2) and which image tag you’re using. - How you’re running Airbyte (Docker Compose vs `abctl`/kind vs Kubernetes/Helm).

The knowledge sources don’t describe building a custom ClickHouse v2 image from tag

2.0.0

specifically, so I can’t say exactly why your build is failing. I can only rely on what’s documented for ClickHouse v2 and local connector development. A few points that are clear from the docs: 1. ClickHouse v2 is the new, GA connector The v2 connector is the actively maintained one and replaces the archived v1 connector. [[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse); [ClickHouse GA](https://github.com/airbytehq/airbyte/discussions/62035)] 2. Recommended way to build a connector locally For local development, the 2.0 platform docs recommend using

airbyte-ci

to build connector images, which produces a

:dev

tag, not `2.0.0`:

bash    airbyte-ci connectors --name destination-clickhouse build    # result: airbyte/destination-clickhouse:dev

[[Connector contributions](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] Then you load that image into kind (if using

abctl

) and set the connector’s Docker tag in Workspace settings → Destinations to

dev

. [[Loading into kind](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#loading-connector-docker-containers-into-kind); [Connector contributions](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] 3. Using

2.0.0

as your build tag The docs don’t show an example of rebuilding ClickHouse v2 with the same semantic version tag (e.g.

2.0.0

) for local development. They consistently use

:dev

as the local tag and then change the tag in the UI. [[Connector contributions](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] So if you’re currently: - Building a local image and tagging it

airbyte/destination-clickhouse:2.0.0

, and - Also pulling

2.0.0

from Docker Hub in some places, there’s a risk of confusion or cache conflicts, but the knowledge sources don’t explicitly document this scenario. --- Given what’s documented, the safest, supported path for your custom changes is: 1. Build your modified ClickHouse destination with

airbyte-ci

and use the default

:dev

tag. 2. Load that image into your cluster (if using kind). 3. In Workspace settings → Destinations, set the Docker image tag for ClickHouse to

dev

. 4. Test your changes. If you must use

2.0.0

as the tag, the knowledge sources don’t provide guidance or guarantees about that workflow, so I can’t confidently recommend it.

I’m sorry, but the knowledge sources do not mention the specific error: >

Failure in destination: Failed to initialize connector operation

for the ClickHouse destination or any other connector, so I can’t tell you exactly which code/config change is required. What is documented for custom / modified destination connectors is how to ensure your custom image is correctly built and used by Airbyte. Misconfiguring these steps can surface as generic “failed to initialize” errors, so it’s worth double‑checking them: --- ### 1. Build the connector image correctly If you’re developing from the Airbyte repo, build the destination from the project root:

bash ./gradlew :airbyte-integrations:connectors:destination-clickhouse:build

This compiles the Java code and builds a Docker image

airbyte/destination-clickhouse:dev

. [[Build Java dest](https://docs.airbyte.com/platform/2.0/connector-development/tutorials/building-a-java-destination#step-2-build-the-newly-generated-destination)] Each time you change the code, you must re‑run this build so the image is updated. [[Build Java dest](https://docs.airbyte.com/platform/2.0/connector-development/tutorials/building-a-java-destination#step-2-build-the-newly-generated-destination)] --- ### 2. Make sure Airbyte can actually run that image #### If you’re using

abctl

(kind / local Kubernetes) You must load the locally built image into the kind cluster:

bash kind load docker-image airbyte/destination-clickhouse:dev -n airbyte-abctl

Otherwise the pod may fail to start or pull the wrong image. [[Load into kind](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#loading-connector-docker-containers-into-kind)] #### If you’re using Docker / Docker Compose You can either: - Use the locally built image directly (no push needed), or - Push it to a registry and then configure Airbyte to pull from there. [[Use custom Docker connector](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)] --- ### 3. Point Airbyte to your custom image/tag In the UI: 1. Go to Workspace settings → Destinations. 2. Either: - Edit the existing ClickHouse destination definition and set the Docker image tag to

dev

(or your custom tag), or - Add a new Docker‑based custom destination with: - `Docker full image name`:

airbyte/destination-clickhouse

- `Docker image tag`:

dev

(or your custom tag). [[Use custom Docker connector](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte); [Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] Now any sync using that destination should run your modified image. --- ### 4. If you changed the spec, refresh the cached spec Airbyte caches connector specifications. If you modified the ClickHouse destination’s

spec

and don’t refresh it, you can get confusing initialization errors. To refresh: 1. Go to Workspace settings → Destinations. 2. Edit the ClickHouse destination. 3. Leave the version/tag set to your dev tag (e.g.

dev

). 4. Click Change to save; this forces a spec refresh. [[Connector spec caching](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-specification-caching)] --- Because the provided docs don’t include this exact error message or ClickHouse‑specific troubleshooting steps, I can’t say which code change is needed. If the error persists after you: - Build

destination-clickhouse

correctly, - Load the image into your environment, - Point Airbyte to the right tag, and - Refresh the spec cache, then the next step (outside the scope of these docs) would be to inspect the job logs for a Java stack trace inside the destination container to see what’s actually failing during initialization.

I'm sorry, but the knowledge sources do not explicitly mention any specific "integration patches" that must be applied to the Airbyte codebase or the Clickhouse connector after building a custom image. However, based on the provided documents, there are a few manual updates and configuration steps you must perform to ensure your custom image is correctly integrated and recognized by your Airbyte instance: ### 1. Register the Custom Image in the UI Once your image is built, you must tell Airbyte to use it. You can either update an existing destination or add a new one: * Navigate to Workspace settings > Destinations. * If you are creating a new entry, click New connector > Add a new Docker connector. * Provide the Docker full image name (e.g.,

airbyte/destination-clickhouse

) and your specific Docker image tag (e.g.,

dev

or your custom version). [[Use custom Docker connector](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte); [Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] ### 2. Force a Specification Refresh If your changes included modifications to the connector's configuration schema (the

spec

), you must clear Airbyte's internal cache: * In the Destinations settings, click the edit button for your connector. * Keep the tag as your custom version and click Change to save. This action forces Airbyte to re-pull the connector's specification. [[Connector spec caching](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-specification-caching)] ### 3. Load Image into Cluster (if using

abctl

) If you are running Airbyte locally via

abctl

, the image exists on your local Docker host but not necessarily inside the Kubernetes cluster. You must load it manually:

bash kind load docker-image <your-image-name>:<tag> -n airbyte-abctl

[[Loading into kind](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#loading-connector-docker-containers-into-kind)] ### 4. Database-level Patch (Specific to ClickHouse V2 Beta) The knowledge sources mention a specific SQL update that was required during the ClickHouse V2 beta to enable certain features like refreshes. While the connector is now GA, if you are working with a custom version of the V2 connector, you may need to run this on your Airbyte metadata database:

sql UPDATE actor_definition_version SET supports_refreshes = true WHERE docker_repository = 'airbyte/destination-clickhouse-v2';

[[ClickHouse V2 guide](https://github.com/airbytehq/airbyte/discussions/62035)] Beyond these configuration steps, the knowledge sources do not contain information on any further code-level "patches" required for the integration to function.

To replace an original connector with your custom-built version in Airbyte, you need to update the connector's configuration in the settings to point to your specific Docker image and tag. Depending on whether you are using a recent version of the platform (v2.0+) or an older version, follow these steps: ### 1. Update the Connector Tag in Settings 1. Log in to your Airbyte instance. 2. Navigate to Workspace settings (the gear icon) and select Sources or Destinations (depending on which connector you are replacing). 3. Find the original connector in the list and click the edit button (or click on the connector name). 4. Locate the field for the Docker image tag. 5. Change the version number to match your custom Docker image tag (e.g.,

dev

or your specific version). 6. Click Change or Save to apply the update [[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions); [Using custom connectors](https://docs.airbyte.com/platform/1.8/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)]. ### 2. Ensure the Image is Accessible * For Local Development (

abctl

): If you built the image locally and are using

abctl

, you must load the image into the local cluster so Airbyte can see it:

bash     kind load docker-image airbyte/source-<name>:<tag> -n airbyte-abctl

[[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-contributions)] * For Remote Deployments: If your Airbyte instance is running on a remote server, you must push your custom image to a private Docker registry that the Airbyte instance is authenticated to pull from [[Using custom connectors](https://docs.airbyte.com/platform/2.0/operator-guides/using-custom-connectors#4-use-your-custom-docker-connector-in-airbyte)]. ### 3. Refresh the Connector Specification If your custom connector has a different configuration schema (specification) than the original, you must force Airbyte to refresh its cache. You can do this by simply clicking Change on the settings page while the version is set to your custom tag; this triggers the Airbyte Server to pull the updated specification from your new image [[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#connector-specification-caching)]. Once these steps are completed, any sync using that connector will automatically use your custom Docker image instead of the official one.