Shubham Singh
06/27/2023, 8:04 AMshubhamsingh@shubhams-MacBook-Pro airbyte % kubectl get pods
NAME READY STATUS RESTARTS AGE
airbyte-airbyte-bootloader 0/1 Error 0 8m7s
airbyte-db-0 1/1 Running 0 8m8s
shubhamsingh@shubhams-MacBook-Pro airbyte %
When i try to check the bootloader logs it shows the following:
2023-06-27 06:44:47 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:90) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:560) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:2861) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchOne(DefaultDSLContext.java:4662) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.lambda$fetchValue$49(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4696) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:4778) ~[jooq-3.17.8.jar:?]
at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:200) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
at io.airbyte.bootloader.Bootloader.load(Bootloader.java:101) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-5.0.1.jar:?]
at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:87) ~[jooq-3.17.8.jar:?]
... 21 more
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "postgres"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693
From the EKS nodes i was able to confirm that the RDS deployed is reachable and the RDS and EKS are in the same VPC. Can you tell me what am i missing here.?
Attaching the values file for your reference. I am using S3 and AWS RDS for storage.Shubham Singh
06/27/2023, 8:50 AMTapan Kumar
01/11/2024, 3:44 PMTapan Kumar
01/11/2024, 3:48 PMdatadog or otel.
2024-01-11 153906 INFO i.a.c.p.v.FeatureFlagDefinitionVersionOverrideProvider(<init>):54 - Initialized feature flag definition version overrides
2024-01-11 153906 INFO i.a.c.p.ActorDefinitionVersionHelper(<init>):68 - ActorDefinitionVersionHelper initialized with override providers: ConfigurationDefinitionVersionOverrideProvider, FeatureFlagDefinitionVersionOverrideProvider
2024-01-11 153906 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable CUSTOMERIO_API_KEY: 'null'
2024-01-11 153906 INFO i.a.b.Bootloader(load):107 - Initializing databases...
2024-01-11 153906 INFO i.a.b.Bootloader(initializeDatabases):232 - Initializing databases...
2024-01-11 153906 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2024-01-11 153906 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
2024-01-11 153936 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:90) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:560) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:2861) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchOne(DefaultDSLContext.java:4662) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.lambda$fetchValue$49(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4696) ~[jooq-3.17.8.jar:?]
at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:4778) ~[jooq-3.17.8.jar:?]
at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47) [io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45) [io.airbyte.airbyte-db-db-lib-0.50.43.jar:?]
at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:233) [io.airbyte-airbyte-bootloader-0.50.43.jar:?]
at io.airbyte.bootloader.Bootloader.load(Bootloader.java:108) [io.airbyte-airbyte-bootloader-0.50.43.jar:?]
at io.airbyte.bootloader.Application.main(Application.java:22) [io.airbyte-airbyte-bootloader-0.50.43.jar:?]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30001ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-5.0.1.jar:?]
at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:87) ~[jooq-3.17.8.jar:?]
... 21 more
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.5.4.jar:42.5.4]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-5.0.1.jar:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.net.UnknownHostException: airbyte-db-svc
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567) ~[?:?]
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
at org.postgresql.core.PGStream.createSocket(PGStream.java:243) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.PGStream.<init>(PGStream.java:98) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.5.4.jar:42.5.4]
at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.5.4.jar:42.5.4]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) ~[HikariCP-5.0.1.jar:?]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-5.0.1.jar:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]
2024-01-11 153936 INFO i.a.d.c.DatabaseAvailabilityCheck(check):49 - Database is not ready yet. Please wait a moment, it might still be initializing...
2024-01-11 153942 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2024-01-11 153942 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
2024-01-11 154012 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)Shubham Singh
01/15/2024, 12:36 PMTapan Kumar
01/16/2024, 3:08 AM