Hi everyone, we've got some problems setting up latest release of Airbyte (0.40.16) inside of an EC2 instance running Ubuntu 22.04. The UI is not reachable and after a while I get a "504 Gateway Time-out". Below are some investigations we did. Where should we look next? We tried

docker-compose down

and

docker-compose up -d

, where the following errors seem to disappear, but still nothing works. The docker images come up nicely.

$ docker-compose ps
       Name                     Command               State                                        Ports                                    
--------------------------------------------------------------------------------------------------------------------------------------------
airbyte-bootloader   /bin/bash -c ${APPLICATION ...   Up                                                                                    
airbyte-cron         /bin/bash -c ${APPLICATION ...   Up                                                                                    
airbyte-db           docker-entrypoint.sh postgres    Up       5432/tcp                                                                     
airbyte-proxy        ./run.sh ./run.sh                Up       80/tcp, 0.0.0.0:8000->8000/tcp,:::8000->8000/tcp,                            
                                                               0.0.0.0:8001->8001/tcp,:::8001->8001/tcp                                     
airbyte-server       /bin/bash -c ${APPLICATION ...   Up       8000/tcp, 0.0.0.0:49316->8001/tcp,:::49316->8001/tcp                         
airbyte-temporal     ./update-and-start-temporal.sh   Up       6933/tcp, 6934/tcp, 6935/tcp, 6939/tcp, 7233/tcp, 7234/tcp, 7235/tcp,        
                                                               7239/tcp                                                                     
airbyte-webapp       /docker-entrypoint.sh ngin ...   Up       0.0.0.0:49317->80/tcp,:::49317->80/tcp                                       
airbyte-worker       /bin/bash -c ${APPLICATION ...   Up       0.0.0.0:49668->9000/tcp,:::49668->9000/tcp                                   
init                 /bin/sh -c ./scripts/creat ...   Exit 0

There are PSQLExceptions in airbyte-worker and airbyte-cron.

$ docker logs airbyte-worker
2022-10-21 11:09:56,409 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
        : airbyte-workers :

  Micronaut (v3.7.2)

2022-10-21 11:09:58 ESC[32mINFOESC[m i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [ec2, cloud, control-plane]
2022-10-21 11:09:58 ESC[32mINFOESC[m c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
2022-10-21 11:10:09 ESC[1;31mERRORESC[m c.z.h.p.HikariPool(throwPoolInitializationException):526 - HikariPool-1 - Exception during pool initialization.
org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:331) ~[postgresql-42.5.0.jar:42.5.0]
       ...
Caused by: java.net.SocketTimeoutException: Connect timed out
        ....

airbyte-server just shows this message at the end of the log:

$ docker logs airbyte-server
2022-10-21 11:09:57 ESC[33mWARNESC[m i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-10-21 11:09:57 ESC[32mINFOESC[m i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...

airbyte-db does not show any errors airbyte-proxy shows some timeouts:

1 upstream timed out (110: Connection timed out) while connecting to upstream

airbyte-temporal

$ docker logs airbyte-temporal
Start init
Done init
Waiting for PostgreSQL to startup.

The database in airbyte-db looks empty 🤔

$ psql -h localhost -p 5432 -d airbyte -U docker
Password for user docker: 
psql (14.5 (Ubuntu 14.5-0ubuntu0.22.04.1), server 13.8)
Type "help" for help.

airbyte=# \d
Did not find any relations.

@Marcos Marx (Airbyte) turned this message into Zendesk ticket 3015 to ensure timely resolution!

Airbyte Proxy (new!) adds HTTP Basic Auth to the application - you may need to add a username/password to your load balancer checks

What load balancer checks? This is just starting Airbyte as described in https://docs.airbyte.com/deploying-airbyte/on-aws-ec2/

@Evan Tahler (Airbyte), do we have any documentation on Airbyte Proxy?

@Saj Dider (Airbyte) nothing specific about how it works other than the README, but we've sprinkled in various docs about changing the auth

Any News on this ? I am also facing same issue.

Which version are you using? I introduced a variable to control timeout for Airbyte proxy, so after .27 it should be working fine