Hello I have k8s cluster placed behind the proxy,...
# replication-troubleshooting
p
Hello I have k8s cluster placed behind the proxy, If I need access to the internet I need to use
HTTP_PROXY
HTTPS_PROXY
NO_PROXY
while trying to deploy airbyte with helm values look like that:
Copy code
webapp:
  env_vars:
    HTTP_PROXY:x
    HTTPS_PROXY:x
    NO_PROXY: x
global:
  env_vars:
    DUMMY: "DUMMY"
server:
  env_vars:
    HTTP_PROXY:x
    HTTPS_PROXY:x
    NO_PROXY: x
worker:
  env_vars:
    HTTP_PROXY:x
    HTTPS_PROXY:x
    NO_PROXY: x
While testing the access to yaml file with curl inside airbyte-server
curl -v <https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/seed/source_definitions.yaml>
Im getting good output but while trying to do something with UI airbyte getting crashed [1] Do I need to add more env_vars with java options to use? https://stackoverflow.com/questions/61658908/java-ignores-http-proxy-settings
Tested with 2 cases:
Copy code
global:
  env_vars:
    DUMMY: "DUMMY"
    -Dhttp.proxyHost: 10.136.0.100
    -Dhttp.proxyPort: 3128
    -Dhttps.proxyHost: 10.136.0.100
    -Dhttps.proxyPort: 3128
    -Dhttp.nonProxyHosts: "10.0.0.0/8|localhost|127.0.0.1|172.20.0.0/16"
And with
Copy code
global:
  env_vars:
    -Djava.net.useSystemProxies: true
Still I’m getting error with warn and timeout
Copy code
│ 2023-01-25 10:27:33 WARN i.a.s.s.AirbyteGithubStore(getLatestDestinations):58 - Unable to retrieve latest Destination list from Github. Using the list bundled with Airbyte. This │
│ java.net.http.HttpConnectTimeoutException: HTTP connect timed out
Overwriting global env_vars with:
Copy code
JAVA_TOOL_OPTIONS: "-Dhttp.proxyHost=10.136.0.100 -Dhttp.proxyPort=000 -Dhttps.proxyHost=10.136.0.100 -Dhttps.proxyPort=000 -Dhttp.nonProxyHosts=10.0.0.0|localhost|127.0.0.1|172.20.0.0|airbyte-*"
Repairs crashes while clicking in the UI and the error about
Unable to retrieve latest Destination list from Github
is not showing up.
Update 2: For proper use of proxy inside worker and spawned jobs, you need to add these configs below
Copy code
worker:
    env_vars:
      JOB_DEFAULT_ENV_http_proxy: <http://10.136.0.000:000>
      JOB_DEFAULT_ENV_https_proxy: <http://10.136.0.000:000>
      JOB_DEFAULT_ENV_no_proxy: "10.0.0.0/8,localhost,127.0.0.1,169.254.169.254,172.20.0.0/16,.internal,<http://s3.amazonaws.com|s3.amazonaws.com>,.<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>,<http://api.ecr.us-east-1.amazonaws.com|api.ecr.us-east-1.amazonaws.com>,<http://dkr.ecr.us-east-1.amazonaws.com|dkr.ecr.us-east-1.amazonaws.com>,<http://ec2.us-east-1.amazonaws.com|ec2.us-east-1.amazonaws.com>,.cluster.local,.local,.svc,airbyte-*"
      JOB_DEFAULT_ENV_HTTP_PROXY: <http://10.136.0.000:000>
      JOB_DEFAULT_ENV_HTTPS_PROXY: <http://10.136.0.000:000>
      JOB_DEFAULT_ENV_NO_PROXY: "10.0.0.0/8,localhost,127.0.0.1,.internal,<http://s3.amazonaws.com|s3.amazonaws.com>,.<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>,<http://api.ecr.us-east-1.amazonaws.com|api.ecr.us-east-1.amazonaws.com>,<http://dkr.ecr.us-east-1.amazonaws.com|dkr.ecr.us-east-1.amazonaws.com>,<http://ec2.us-east-1.amazonaws.com|ec2.us-east-1.amazonaws.com>,.cluster.local,.local,.svc,airbyte-*"
      JOB_DEFAULT_ENV_JAVA_TOOL_OPTIONS: "-Dhttp.proxyHost=10.136.0.000 -Dhttp.proxyPort=000 -Dhttps.proxyHost=10.136.0.000 -Dhttps.proxyPort=000 -Dhttp.nonProxyHosts=10.0.0.0|localhost|127.0.0.1|*.internal|<http://s3.amazonaws.com|s3.amazonaws.com>|*.<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>|<http://api.ecr.us-east-1.amazonaws.com|api.ecr.us-east-1.amazonaws.com>|<http://dkr.ecr.us-east-1.amazonaws.com|dkr.ecr.us-east-1.amazonaws.com>|<http://ec2.us-east-1.amazonaws.com|ec2.us-east-1.amazonaws.com>|*.cluster.local|*.local|*.svc|airbyte-*"
@Marcos Marx (Airbyte) Hey sorry for pinging you directly, I didn’t find any material about using proxy with airbyte, what do you think about creating issue or some docs about how to properly configure airbyte values to use proxy?
m
Hello @Piotr Sierkin sorry the delay to reply to you. I’ll take a look tomorrow morning!
❤️ 1
p
k
hi there, thanks to this post, i was able to fix similar issues when running airbyte behind a corporate proxy. It is very important to not only specify env variables like
HTTP_PROXY
but also the ones used by the JVM. I had to adapt a bit what's above but it essentially worked
b
@Kevin Serrano would you be able to share any insight on what you had to adapt? I'm trying to adapt this to a docker-compose deployment but with not success yet.
k
hi @Bruno Pini, I had to adapt the values of the proxy and no_proxy to match my needs. Here is the config i used (in helm chart
values.yaml
) but i have redacted some values. The trick is that you have to make sure the ENV variable are passed to the JVM, which is why i used the
JAVA_TOOL_OPTIONS
. using
JOB_DEFAULT_ENV_*
is to make sure the variables are passed to the pods that are created by the worker service (during sync for example). Not sure if it's the same for docker-compose. I also had to inject the CA chain of the proxy into the pod otherwise i would get a SSL error because it could not validate it. You can test this by using openssl or just even
curl
from the worker pod
Copy code
- name: JOB_DEFAULT_ENV_HTTP_PROXY
        value: <http://myproxy:80>
      - name: JOB_DEFAULT_ENV_HTTPS_PROXY
        value: <https://myproxy:443>
      - name: JOB_DEFAULT_ENV_NO_PROXY
        value: .<http://rds.amazonaws.com|rds.amazonaws.com>,.<http://myinternaldomain.net|myinternaldomain.net>,localhost,10.0.0.0/8,172.16.0.0/21,192.168.0.0/16,kubernetes,.cluster.local,.local,.svc,airbyte-minio-svc,airbyte-dev-airbyte-api-server-svc,airbyte-dev-airbyte-server-svc,airbyte-dev-airbyte-webapp-svc,airbyte-dev-temporal
      - name: JOB_DEFAULT_ENV_JAVA_TOOL_OPTIONS
        value: "-Dhttp.proxyHost=myproxy -Dhttp.proxyPort=80 -Dhttps.proxyHost=myproxy -Dhttps.proxyPort=443 -Dhttp.nonProxyHosts=*<http://rds.amazonaws.com|rds.amazonaws.com>|*.<http://myinternaldomain.net|myinternaldomain.net>|localhost|10.*|172.16.*|192.168.*|kubernetes|*.local|airbyte-minio-svc|airbyte-dev-airbyte-api-server-svc|airbyte-dev-airbyte-server-svc|airbyte-dev-airbyte-webapp-svc|airbyte-dev-temporal"
      - name: HTTP_PROXY
        value: <http://myproxy:80>
      - name: HTTPS_PROXY
        value: <https://myproxy:443>
      - name: NO_PROXY
        value: .<http://rds.amazonaws.com|rds.amazonaws.com>,.<http://myinternaldomain.net|myinternaldomain.net>,localhost,10.0.0.0/8,172.16.0.0/21,192.168.0.0/16,kubernetes,.cluster.local,.local,.svc,airbyte-minio-svc,airbyte-dev-airbyte-api-server-svc,airbyte-dev-airbyte-server-svc,airbyte-dev-airbyte-webapp-svc,airbyte-dev-temporal
      - name: JAVA_TOOL_OPTIONS
        value: "-Dhttp.proxyHost=myproxy -Dhttp.proxyPort=80 -Dhttps.proxyHost=myproxy -Dhttps.proxyPort=443 -Dhttp.nonProxyHosts=*<http://rds.amazonaws.com|rds.amazonaws.com>|*.<http://myinternaldomain.net|myinternaldomain.net>|localhost|10.*|172.16.*|192.168.*|kubernetes|*.local|airbyte-minio-svc|airbyte-dev-airbyte-api-server-svc|airbyte-dev-airbyte-server-svc|airbyte-dev-airbyte-webapp-svc|airbyte-dev-temporal"