hello, I am trying to install Airbyte using offici...
# replication-ask-ai
s
hello, I am trying to install Airbyte using official helm chart values provided. I am seeing that the bottloader is in error status:
Copy code
shubhamsingh@shubhams-MacBook-Pro airbyte % kubectl get pods                                                            
NAME                         READY   STATUS    RESTARTS   AGE
airbyte-airbyte-bootloader   0/1     Error     0          8m7s
airbyte-db-0                 1/1     Running   0          8m8s
shubhamsingh@shubhams-MacBook-Pro airbyte %
When i try to check the bootloader logs it shows the following:
Copy code
2023-06-27 06:44:47 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection.
org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
	at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:90) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:560) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:2861) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchOne(DefaultDSLContext.java:4662) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.lambda$fetchValue$49(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4696) ~[jooq-3.17.8.jar:?]
	at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:4778) ~[jooq-3.17.8.jar:?]
	at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45) ~[io.airbyte.airbyte-db-db-lib-0.50.1.jar:?]
	at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:200) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
	at io.airbyte.bootloader.Bootloader.load(Bootloader.java:101) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
	at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.50.1.jar:?]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) ~[HikariCP-5.0.1.jar:?]
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-5.0.1.jar:?]
	at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:87) ~[jooq-3.17.8.jar:?]
	... 21 more
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user "postgres"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:693
From the EKS nodes i was able to confirm that the RDS deployed is reachable and the RDS and EKS are in the same VPC. Can you tell me what am i missing here.? Attaching the values file for your reference. I am using S3 and AWS RDS for storage.
Never mind.. This is resolved.. there was an issue with the helm chart..
t
Hi @Shubham Singh could you please explain what was the issue you faced with the helm chart and how did you fix it I'm facing the same issue. even I tried with the latest chart version and it's getting stuck at the bootloader pod when I further debug it then it shows that the DB server was not able to provision
Here is the error message I can see with the bootloader pod. root@master:/home/ubuntu# kubectl logs airbyte-airbyte-bootloader INFO StatusConsoleListener Loading mask data from '/seed/specs_secrets_mask.yaml _ _ _ _ / | (_)____/ /_ __ __/ /____ / /| | / / ___/ __ \/ / / / __/ _ \ / _ |/ / / / /_/ / /_/ / /_/ __/ /_/ |_/_/_/ /_.___/\__, /\__/\___/ /____/ : airbyte-bootloader : Micronaut (v3.10.1) 2024-01-11 153903 INFO i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [k8s, cloud, ec2] 2024-01-11 153904 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 2245ms. Server Running: http://airbyte-airbyte-bootloader:9002 2024-01-11 153905 INFO i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values 2024-01-11 153905 INFO i.a.c.s.RemoteDefinitionsProvider(<init>):74 - Creating remote definitions provider for URL 'https://connectors.airbyte.com/files/' and registry 'OSS'... 2024-01-11 153905 INFO i.a.c.i.c.SeedBeanFactory(seedDefinitionsProvider):37 - Using local definitions provider for seeding 2024-01-11 153905 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable LOCAL_CONNECTOR_CATALOG_PATH: 'seed/local_oss_registry.json' 2024-01-11 153905 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable METRIC_CLIENT: '' 2024-01-11 153905 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable METRIC_CLIENT: '' 2024-01-11 153905 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are
datadog
or
otel
. 2024-01-11 153906 INFO i.a.c.p.v.FeatureFlagDefinitionVersionOverrideProvider(<init>):54 - Initialized feature flag definition version overrides 2024-01-11 153906 INFO i.a.c.p.ActorDefinitionVersionHelper(<init>):68 - ActorDefinitionVersionHelper initialized with override providers: ConfigurationDefinitionVersionOverrideProvider, FeatureFlagDefinitionVersionOverrideProvider 2024-01-11 153906 INFO i.a.c.EnvConfigs(getEnvOrDefault):1149 - Using default value for environment variable CUSTOMERIO_API_KEY: 'null' 2024-01-11 153906 INFO i.a.b.Bootloader(load):107 - Initializing databases... 2024-01-11 153906 INFO i.a.b.Bootloader(initializeDatabases):232 - Initializing databases... 2024-01-11 153906 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available... 2024-01-11 153906 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection... 2024-01-11 153936 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection. org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2) at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?] at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:90) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultExecuteContext.connection(DefaultExecuteContext.java:560) ~[jooq-3.17.8.jar:?] at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:296) ~[jooq-3.17.8.jar:?] at org.jooq.impl.AbstractResultQuery.fetchLazy(AbstractResultQuery.java:301) ~[jooq-3.17.8.jar:?] at org.jooq.impl.AbstractResultQuery.fetchLazyNonAutoClosing(AbstractResultQuery.java:322) ~[jooq-3.17.8.jar:?] at org.jooq.impl.SelectImpl.fetchLazyNonAutoClosing(SelectImpl.java:2861) ~[jooq-3.17.8.jar:?] at org.jooq.impl.ResultQueryTrait.fetchOne(ResultQueryTrait.java:509) ~[jooq-3.17.8.jar:?] at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.fetchOne(DefaultDSLContext.java:4662) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.lambda$fetchValue$49(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?] at org.jooq.impl.Tools.attach(Tools.java:1526) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4682) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.fetchValue(DefaultDSLContext.java:4696) ~[jooq-3.17.8.jar:?] at org.jooq.impl.DefaultDSLContext.fetchExists(DefaultDSLContext.java:4778) ~[jooq-3.17.8.jar:?] at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$0(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?] at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?] at io.airbyte.db.check.DatabaseAvailabilityCheck.lambda$isDatabaseConnected$1(DatabaseAvailabilityCheck.java:76) ~[io.airbyte.airbyte-db-db-lib-0.50.43.jar:?] at io.airbyte.db.check.DatabaseAvailabilityCheck.check(DatabaseAvailabilityCheck.java:47) [io.airbyte.airbyte-db-db-lib-0.50.43.jar:?] at io.airbyte.db.init.DatabaseInitializer.initialize(DatabaseInitializer.java:45) [io.airbyte.airbyte-db-db-lib-0.50.43.jar:?] at io.airbyte.bootloader.Bootloader.initializeDatabases(Bootloader.java:233) [io.airbyte-airbyte-bootloader-0.50.43.jar:?] at io.airbyte.bootloader.Bootloader.load(Bootloader.java:108) [io.airbyte-airbyte-bootloader-0.50.43.jar:?] at io.airbyte.bootloader.Application.main(Application.java:22) [io.airbyte-airbyte-bootloader-0.50.43.jar:?] Caused by: java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30001ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[HikariCP-5.0.1.jar:?] at org.jooq.impl.DataSourceConnectionProvider.acquire(DataSourceConnectionProvider.java:87) ~[jooq-3.17.8.jar:?] ... 21 more Caused by: org.postgresql.util.PSQLException: The connection attempt failed. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:354) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.5.4.jar:42.5.4] at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-5.0.1.jar:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?] Caused by: java.net.UnknownHostException: airbyte-db-svc at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:567) ~[?:?] at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?] at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?] at org.postgresql.core.PGStream.createSocket(PGStream.java:243) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.core.PGStream.<init>(PGStream.java:98) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:132) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:258) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:253) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.Driver.makeConnection(Driver.java:434) ~[postgresql-42.5.4.jar:42.5.4] at org.postgresql.Driver.connect(Driver.java:291) ~[postgresql-42.5.4.jar:42.5.4] at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:733) ~[HikariCP-5.0.1.jar:?] at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-5.0.1.jar:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?] 2024-01-11 153936 INFO i.a.d.c.DatabaseAvailabilityCheck(check):49 - Database is not ready yet. Please wait a moment, it might still be initializing... 2024-01-11 153942 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available... 2024-01-11 153942 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection... 2024-01-11 154012 ERROR i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):78 - Failed to verify database connection. org.jooq.exception.DataAccessException: Error getting connection from data source HikariDataSource (HikariPool-2)
s
hi tapan, for me airbyte-db did not exist on the rds, i created it manually, then this issue was resolved..
👍 1
t
for me it's available on the same cluster and even DB pod and DB service deployed successfully still it's not able to resolve DB host address whereas if I deploy a test pod into the same cluster then it's able to resolve DB host address using DB service name
👀 1