Piotr Sierkin
01/25/2023, 9:08 AMHTTP_PROXY HTTPS_PROXY NO_PROXY while trying to deploy airbyte with helm values look like that:
webapp:
env_vars:
HTTP_PROXY:x
HTTPS_PROXY:x
NO_PROXY: x
global:
env_vars:
DUMMY: "DUMMY"
server:
env_vars:
HTTP_PROXY:x
HTTPS_PROXY:x
NO_PROXY: x
worker:
env_vars:
HTTP_PROXY:x
HTTPS_PROXY:x
NO_PROXY: x
While testing the access to yaml file with curl inside airbyte-server
curl -v <https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/seed/source_definitions.yaml> Im getting good output but while trying to do something with UI airbyte getting crashed [1]
Do I need to add more env_vars with java options to use? https://stackoverflow.com/questions/61658908/java-ignores-http-proxy-settingsPiotr Sierkin
01/25/2023, 10:27 AMglobal:
env_vars:
DUMMY: "DUMMY"
-Dhttp.proxyHost: 10.136.0.100
-Dhttp.proxyPort: 3128
-Dhttps.proxyHost: 10.136.0.100
-Dhttps.proxyPort: 3128
-Dhttp.nonProxyHosts: "10.0.0.0/8|localhost|127.0.0.1|172.20.0.0/16"
And with
global:
env_vars:
-Djava.net.useSystemProxies: true
Still I’m getting error with warn and timeout
│ 2023-01-25 10:27:33 WARN i.a.s.s.AirbyteGithubStore(getLatestDestinations):58 - Unable to retrieve latest Destination list from Github. Using the list bundled with Airbyte. This │
│ java.net.http.HttpConnectTimeoutException: HTTP connect timed outPiotr Sierkin
01/26/2023, 8:57 AMJAVA_TOOL_OPTIONS: "-Dhttp.proxyHost=10.136.0.100 -Dhttp.proxyPort=000 -Dhttps.proxyHost=10.136.0.100 -Dhttps.proxyPort=000 -Dhttp.nonProxyHosts=10.0.0.0|localhost|127.0.0.1|172.20.0.0|airbyte-*"
Repairs crashes while clicking in the UI and the error about Unable to retrieve latest Destination list from Github is not showing up.Piotr Sierkin
01/26/2023, 11:14 AMworker:
env_vars:
JOB_DEFAULT_ENV_http_proxy: <http://10.136.0.000:000>
JOB_DEFAULT_ENV_https_proxy: <http://10.136.0.000:000>
JOB_DEFAULT_ENV_no_proxy: "10.0.0.0/8,localhost,127.0.0.1,169.254.169.254,172.20.0.0/16,.internal,<http://s3.amazonaws.com|s3.amazonaws.com>,.<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>,<http://api.ecr.us-east-1.amazonaws.com|api.ecr.us-east-1.amazonaws.com>,<http://dkr.ecr.us-east-1.amazonaws.com|dkr.ecr.us-east-1.amazonaws.com>,<http://ec2.us-east-1.amazonaws.com|ec2.us-east-1.amazonaws.com>,.cluster.local,.local,.svc,airbyte-*"
JOB_DEFAULT_ENV_HTTP_PROXY: <http://10.136.0.000:000>
JOB_DEFAULT_ENV_HTTPS_PROXY: <http://10.136.0.000:000>
JOB_DEFAULT_ENV_NO_PROXY: "10.0.0.0/8,localhost,127.0.0.1,.internal,<http://s3.amazonaws.com|s3.amazonaws.com>,.<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>,<http://api.ecr.us-east-1.amazonaws.com|api.ecr.us-east-1.amazonaws.com>,<http://dkr.ecr.us-east-1.amazonaws.com|dkr.ecr.us-east-1.amazonaws.com>,<http://ec2.us-east-1.amazonaws.com|ec2.us-east-1.amazonaws.com>,.cluster.local,.local,.svc,airbyte-*"
JOB_DEFAULT_ENV_JAVA_TOOL_OPTIONS: "-Dhttp.proxyHost=10.136.0.000 -Dhttp.proxyPort=000 -Dhttps.proxyHost=10.136.0.000 -Dhttps.proxyPort=000 -Dhttp.nonProxyHosts=10.0.0.0|localhost|127.0.0.1|*.internal|<http://s3.amazonaws.com|s3.amazonaws.com>|*.<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>|<http://api.ecr.us-east-1.amazonaws.com|api.ecr.us-east-1.amazonaws.com>|<http://dkr.ecr.us-east-1.amazonaws.com|dkr.ecr.us-east-1.amazonaws.com>|<http://ec2.us-east-1.amazonaws.com|ec2.us-east-1.amazonaws.com>|*.cluster.local|*.local|*.svc|airbyte-*"Piotr Sierkin
01/26/2023, 11:15 AMMarcos Marx (Airbyte)
01/30/2023, 10:53 PMPiotr Sierkin
01/31/2023, 10:07 AMKevin Serrano
05/13/2024, 6:46 AMHTTP_PROXY but also the ones used by the JVM. I had to adapt a bit what's above but it essentially workedBruno Pini
05/14/2024, 6:12 PMKevin Serrano
05/15/2024, 7:38 AMvalues.yaml) but i have redacted some values. The trick is that you have to make sure the ENV variable are passed to the JVM, which is why i used the JAVA_TOOL_OPTIONS . using JOB_DEFAULT_ENV_* is to make sure the variables are passed to the pods that are created by the worker service (during sync for example). Not sure if it's the same for docker-compose. I also had to inject the CA chain of the proxy into the pod otherwise i would get a SSL error because it could not validate it. You can test this by using openssl or just even curl from the worker pod
- name: JOB_DEFAULT_ENV_HTTP_PROXY
value: <http://myproxy:80>
- name: JOB_DEFAULT_ENV_HTTPS_PROXY
value: <https://myproxy:443>
- name: JOB_DEFAULT_ENV_NO_PROXY
value: .<http://rds.amazonaws.com|rds.amazonaws.com>,.<http://myinternaldomain.net|myinternaldomain.net>,localhost,10.0.0.0/8,172.16.0.0/21,192.168.0.0/16,kubernetes,.cluster.local,.local,.svc,airbyte-minio-svc,airbyte-dev-airbyte-api-server-svc,airbyte-dev-airbyte-server-svc,airbyte-dev-airbyte-webapp-svc,airbyte-dev-temporal
- name: JOB_DEFAULT_ENV_JAVA_TOOL_OPTIONS
value: "-Dhttp.proxyHost=myproxy -Dhttp.proxyPort=80 -Dhttps.proxyHost=myproxy -Dhttps.proxyPort=443 -Dhttp.nonProxyHosts=*<http://rds.amazonaws.com|rds.amazonaws.com>|*.<http://myinternaldomain.net|myinternaldomain.net>|localhost|10.*|172.16.*|192.168.*|kubernetes|*.local|airbyte-minio-svc|airbyte-dev-airbyte-api-server-svc|airbyte-dev-airbyte-server-svc|airbyte-dev-airbyte-webapp-svc|airbyte-dev-temporal"
- name: HTTP_PROXY
value: <http://myproxy:80>
- name: HTTPS_PROXY
value: <https://myproxy:443>
- name: NO_PROXY
value: .<http://rds.amazonaws.com|rds.amazonaws.com>,.<http://myinternaldomain.net|myinternaldomain.net>,localhost,10.0.0.0/8,172.16.0.0/21,192.168.0.0/16,kubernetes,.cluster.local,.local,.svc,airbyte-minio-svc,airbyte-dev-airbyte-api-server-svc,airbyte-dev-airbyte-server-svc,airbyte-dev-airbyte-webapp-svc,airbyte-dev-temporal
- name: JAVA_TOOL_OPTIONS
value: "-Dhttp.proxyHost=myproxy -Dhttp.proxyPort=80 -Dhttps.proxyHost=myproxy -Dhttps.proxyPort=443 -Dhttp.nonProxyHosts=*<http://rds.amazonaws.com|rds.amazonaws.com>|*.<http://myinternaldomain.net|myinternaldomain.net>|localhost|10.*|172.16.*|192.168.*|kubernetes|*.local|airbyte-minio-svc|airbyte-dev-airbyte-api-server-svc|airbyte-dev-airbyte-server-svc|airbyte-dev-airbyte-webapp-svc|airbyte-dev-temporal"