<@U04S8PLJWMT> I understand that it's currently di...
# ask-ai
h
@kapa.ai I understand that it's currently difficult to operate Airbyte correctly in a proxy environment. If I want to build Airbyte in a zone that cannot directly access the internet, what are the steps I would need to follow for deployment? Are there any alternative solutions, such as setting up a local repository by building our own server as a mirror server, and having Airbyte reference that?
k
Running Airbyte in a zone without direct internet access is currently not officially supported as a fully offline scenario, and multiple reports show that Airbyte requires internet connectivity (or at least access to specific remote resources) to function correctly. ### 1. What is required from the network side? From the shared materials: - Airbyte OSS requires internet access to fetch resources from GitHub (for connector definitions and other assets). [[Issue GKE no‑internet](https://discuss.airbyte.io/t/763); [No‑internet docker deploy](https://discuss.airbyte.io/t/4042); [Install without internet](https://discuss.airbyte.io/t/4116)] - Specifically, the server/bootloader fetches: -
source_definitions.yaml
-
destination_definitions.yaml
from GitHub. [[Issue GKE no‑internet](https://discuss.airbyte.io/t/763)] - Without any internet, current community answers state “it is currently not possible to use Airbyte without an internet connection.” [[Install without internet](https://discuss.airbyte.io/t/4116)] So you must provide some path (direct, via proxy, or via an internal mirror) for Airbyte to reach those resources. ### 2. Proxy support and current limitations The Helm-based deployment behind an authenticated proxy is known to be problematic: - Users report failures when the environment requires an authenticating proxy; even with
HTTPS_PROXY
,
NO_PROXY
, and
JAVA_TOOL_OPTIONS
set, the bootloader fails to authenticate to the proxy while trying to reach
<https://connectors.airbyte.com>
. [[Proxy auth Helm 1.4.1](https://github.com/airbytehq/airbyte/issues/53227)] - Another user notes that Airbyte Helm charts historically didn’t clearly support proxy configuration, and they considered falling back to Cloud NAT. [[Issue GKE no‑internet](https://discuss.airbyte.io/t/763)] - From 1.1.0+, proxy env vars in
global.env_vars
are not propagated to sidecar containers used in check/sync jobs, so even if core pods see the proxy, some task containers still won’t. [[Proxy for sidecar](https://github.com/airbytehq/airbyte/issues/56917)] - Even adding the Helm repo itself can be blocked by corporate proxies returning
403
to
<https://airbytehq.github.io/helm-charts>
. [[Helm repo 403](https://github.com/airbytehq/airbyte/issues/50840)] Given these, running purely via an enterprise proxy is possible only with careful tuning and may still hit product limitations. ### 3. Pattern used by other users in restricted networks One concrete workaround (not fully offline, but “internet via a controlled internal mirror”) was documented: - A user whose Kubernetes cluster had no public internet access: 1. Manually downloaded the connector registry JSON file. 2. Stood up a small internal HTTP server that serves this JSON. 3. Pointed Airbyte to that internal URL by setting the environment variable
CONNECTOR_REGISTRY_BASE_URL
. 4. Upgraded Airbyte successfully afterwards. [[Connector registry mirror](https://github.com/airbytehq/airbyte/issues/51511)] This shows that at least some of the “internet dependencies” can be redirected to an internal mirror. However, the knowledge sources do not contain a complete, official procedure for mirroring all external calls (e.g., GitHub seed YAMLs,
<http://connectors.airbyte.com|connectors.airbyte.com>
assets, Helm repo, etc.). So the following is what is explicitly described plus what is reasonably implied by those examples. ### 4. Practical deployment approach in a no-direct-internet zone Based on the issues and forum posts, you have two main patterns: #### A. “Semi‑offline” with outbound proxy / NAT If your zone is allowed to reach the internet via a central egress (proxy or NAT), this is the simplest supported pattern: 1. Provide outbound egress (NAT or HTTP(S) proxy) for: -
<https://airbytehq.github.io/helm-charts>
(for Helm-based install/upgrade). [[Helm repo 403](https://github.com/airbytehq/airbyte/issues/50840)] -
<https://connectors.airbyte.com>
(connector registry / metadata). [[Proxy auth Helm 1.4.1](https://github.com/airbytehq/airbyte/issues/53227)] - GitHub raw content URLs for seed definitions. [[Issue GKE no‑internet](https://discuss.airbyte.io/t/763)] 2. Configure Airbyte pods to use the proxy: - Set
HTTP_PROXY
,
HTTPS_PROXY
,
NO_PROXY
(and lower‑case equivalents) in
values.yaml
under
global.env_vars
so core components get them. [[Proxy for sidecar](https://github.com/airbytehq/airbyte/issues/56917)] - Be aware that (per 1.1.0+ issue) these env vars are not propagated to all sidecar containers, which may still block some operations. [[Proxy for sidecar](https://github.com/airbytehq/airbyte/issues/56917)] - For Java processes, some users also set
JAVA_TOOL_OPTIONS
with
-Dhttps.proxyHost
,
-Dhttps.proxyPort
,
-Dhttps.proxyUser
,
-Dhttps.proxyPassword
, but there are reports of authentication still failing. [[Proxy auth Helm 1.4.1](https://github.com/airbytehq/airbyte/issues/53227)] 3. Accept that this pattern can still hit bugs: - The “Deployment Fails when proxy setting requires Authentication” issue is still open and considered likely a software bug. [[Proxy auth Helm 1.4.1](https://github.com/airbytehq/airbyte/issues/53227)] - You may need to allow non‑auth outbound proxy for Airbyte, or use NAT instead of an authenticating proxy when possible. #### B. Internal mirror (“local repository”) pattern This is closest to what you asked (local repository / mirror server). From available information, only part is documented: 1. Mirror connector registry: - Download the registry JSON file(s) that Airbyte normally fetches (the exact file name isn’t in the provided sources, but one user describes “manually downloading the json file” used by bootloader). [[Connector registry mirror](https://github.com/airbytehq/airbyte/issues/51511)] - Host them on an internal HTTP server reachable from your cluster. 2. Point Airbyte at the internal mirror: - Set the environment variable
CONNECTOR_REGISTRY_BASE_URL
to your internal mirror URL; this allowed that user to upgrade without public internet. [[Connector registry mirror](https://github.com/airbytehq/airbyte/issues/51511)] 3. Limitations / unknown parts: - The knowledge sources do not document all URLs that Airbyte may call (e.g., whether the seed YAMLs from GitHub can be redirected similarly or must still be reachable externally). - There is no official guidance on mirroring
<https://airbytehq.github.io/helm-charts>
or
<https://connectors.airbyte.com>
as a whole; only the registry JSON redirect via
CONNECTOR_REGISTRY_BASE_URL
is explicitly mentioned. [[Connector registry mirror](https://github.com/airbytehq/airbyte/issues/51511)] - Forum answers state clearly that running with zero internet access is currently “not possible,” which implies mirrors alone are not documented as a complete solution. [[Install without internet](https://discuss.airbyte.io/t/4116)] Given that, a realistic design in a very restricted network is often: - Cluster has no direct internet, but - Images are pre‑pulled into an internal registry (as one user did with JFrog). [[Issue GKE no‑internet](https://discuss.airbyte.io/t/763)] - A very small, controlled egress mechanism exists (NAT or proxy) to only allow: - GitHub raw URLs for seed files -
<http://connectors.airbyte.com|connectors.airbyte.com>
(or an internal mirror reachable via
CONNECTOR_REGISTRY_BASE_URL
) - Helm chart repo, if using Helm directly ### 5. Fully offline deployment (no egress at all) Based on the documents: - Community support explicitly states: “it is currently not possible to use Airbyte without an internet connection.” [[Install without internet](https://discuss.airbyte.io/t/4116)] - Another thread confirms Airbyte needs internet access to retrieve the
source_definitions.yaml
and
destination_definitions.yaml
files from GitHub. [[Issue GKE no‑internet](https://discuss.airbyte.io/t/763)] - There is no official documentation in the provided sources describing a fully supported “air‑gapped” procedure where all of these dependencies are mirrored locally and Airbyte is run completely offline. So, with the information available, a fully air‑gapped, no‑internet deployment is not supported, and there is no complete, officially documented mirror‑server pattern beyond redirecting
CONNECTOR_REGISTRY_BASE_URL
as a workaround. --- Summary - Airbyte requires some internet access for connector definitions and related resources; running with zero internet is currently not supported. [[Install without internet](https://discuss.airbyte.io/t/4116); [Issue GKE no‑internet](https://discuss.airbyte.io/t/763)] - Authenticated proxy support via Helm is incomplete and has known issues; sidecar containers may not receive proxy env vars. [[Proxy auth Helm 1.4.1](https://github.com/airbytehq/airbyte/issues/53227); [Proxy for sidecar](https://github.com/airbytehq/airbyte/issues/56917)] - A partial “local repository” pattern is possible by: - Manually downloading the connector registry JSON; - Serving it internally; - Pointing Airbyte to it with
CONNECTOR_REGISTRY_BASE_URL
. [[Connector registry mirror](https://github.com/airbytehq/airbyte/issues/51511)] - Beyond this, the knowledge sources don’t document a full mirror strategy, so any deeper air‑gapped design would go beyond what is officially described.