https://linen.dev logo
Join Slack
Powered by
Hi all, We're upgrading our Airbyte deployment fr...
# replication-ask-ai
w
Hi all, We're upgrading our Airbyte deployment from version 0.40.32 to version 0.50.30. This is a docker deployment on an AWS EC2 instance with the config database being an AWS RDS instance. Following the upgrade instructions here for Docker upgrades, upgrading it not using the 
./run-ab-platform.sh
script but just 
docker compose down; docker compose up
, and also deploying the new version on a Kubernetes cluster via helm and pointing that at the RDS instance, these all result in the same error during the run of the bootloader container 
i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
The bootloader container can connect to the RDS instance, and the db user has full read/write/admin permissions over the RDS instance, but produces a 
org.jooq.exception.DataAccessException
as it cannot read the table 
public.actor_definition_version
due to it not existing on the Airbyte database on the RDS instance. On a separate Airbyte 0.50.30 Kubernetes deployment, I can inspect the 
airbyte
database and see that there are 39 tables within it, compared to 23 tables within the 
airbyte
database from our EC2 Docker deployment in version 0.40.32. The error seems to only say that it cannot access the table because it doesn't exist. Not that it cannot create the tables. I would imagine during the migration step it should create any tables which did not exist in previous versions? I don't want to manually create the tables as that would likely cause more issues later on, and would be a painfully slow process. Any tips? Has anyone encountered this before, and if so how did you fix it? Full logs pasted in reply to this message, as not to pollute this chat.
Copy code
Attaching to airbyte-api-server, airbyte-bootloader, airbyte-connector-builder-server, airbyte-cron, airbyte-db, airbyte-proxy, airbyte-server, airbyte-temporal, airbyte-webapp, airbyte-worker, init
init                              | MOUNT: /local_parent
init                              | ROOT_PARENT: /tmp
init                              | ROOT: /tmp/airbyte_local
init                              | MOUNT_ROOT: /local_parent//airbyte_local
init exited with code 0
airbyte-bootloader                | 2023-10-13 16:13:09,719 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml
airbyte-bootloader                |
airbyte-bootloader                |     ___    _      __          __
airbyte-bootloader                |    /   |  (_)____/ /_  __  __/ /____
airbyte-bootloader                |   / /| | / / ___/ __ \/ / / / __/ _ \
airbyte-bootloader                |  / ___ |/ / /  / /_/ / /_/ / /_/  __/
airbyte-bootloader                | /_/  |_/_/_/  /_.___/\__, /\__/\___/
airbyte-bootloader                |                     /____/
airbyte-bootloader                |         : airbyte-bootloader :
airbyte-bootloader                |
airbyte-bootloader                |
airbyte-bootloader                |   Micronaut (v3.10.1)
airbyte-bootloader                |
airbyte-bootloader                |
airbyte-bootloader                | 2023-10-13 16:13:11 INFO i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [ec2, cloud]
airbyte-bootloader                | 2023-10-13 16:13:12 INFO i.m.r.Micronaut(lambda$start$2):98 - Startup completed in 2255ms. Server Running: <http://324fec77856d:9002>
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.f.ConfigFileClient(<init>):96 - path /flags does not exist, will return default flag values
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.c.s.RemoteDefinitionsProvider(<init>):74 - Creating remote definitions provider for URL '<https://connectors.airbyte.com/files/>' and registry 'OSS'...
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.c.i.c.SeedBeanFactory(seedDefinitionsProvider):37 - Using local definitions provider for seeding
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.c.EnvConfigs(getEnvOrDefault):1228 - Using default value for environment variable LOCAL_CONNECTOR_CATALOG_PATH: 'seed/oss_registry.json'
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.c.p.v.DefaultDefinitionVersionOverrideProvider(<init>):52 - Initialized feature flag definition version overrides
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.c.p.ActorDefinitionVersionHelper(<init>):57 - ActorDefinitionVersionHelper initialized with override provider: DefaultDefinitionVersionOverrideProvider
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.b.Bootloader(load):99 - Initializing databases...
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.b.Bootloader(initializeDatabases):201 - Initializing databases...
airbyte-bootloader                | 2023-10-13 16:13:13 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
airbyte-bootloader                | 2023-10-13 16:13:13 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte configs database is initialized
airbyte-bootloader                | 2023-10-13 16:13:14 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte jobs database connection...
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte jobs database is initialized
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.Bootloader(initializeDatabases):204 - Databases initialized.
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.Bootloader(load):102 - Checking migration compatibility...
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.Bootloader(assertNonBreakingMigration):134 - Checking for illegal upgrade...
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.Bootloader(getRequiredVersionUpgrade):215 - Current Airbyte version: AirbyteVersion{version='0.40.32', major='0', minor='40', patch='32'}
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.Bootloader(getRequiredVersionUpgrade):216 - Future Airbyte version: AirbyteVersion{version='0.50.30', major='0', minor='50', patch='30'}
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.Bootloader(load):105 - Checking protocol version constraints...
airbyte-bootloader                | 2023-10-13 16:13:14 INFO i.a.b.ProtocolVersionChecker(validate):92 - Detected an AirbyteProtocolVersion range change from [0.0.0:1.0.0] to [0.0.0:0.3.0]
airbyte-bootloader                | 2023-10-13 16:13:14 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
airbyte-bootloader                | org.jooq.exception.DataAccessException: SQL [select distinct "public"."actor_definition"."id", "public"."actor_definition_version"."docker_repository", "public"."actor_definition"."actor_type", "public"."actor_definition_version"."protocol_version" from "public"."actor_definition" join "public"."actor" on "public"."actor"."actor_definition_id" = "public"."actor_definition"."id" join "public"."actor_definition_version" on "public"."actor_definition_version"."id" = "public"."actor_definition"."default_version_id"]; ERROR: relation "public.actor_definition_version" does not exist
airbyte-bootloader                |   Position: 342
airbyte-bootloader                | 	at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
airbyte-bootloader                | 	at org.jooq.impl.Tools.translate(Tools.java:3313) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:678) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:355) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:290) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.SelectImpl.fetch(SelectImpl.java:2837) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ConfigRepository.getActorDefinitionsInUse(ConfigRepository.java:649) ~[io.airbyte.airbyte-config-config-persistence-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ConfigRepository.lambda$getActorDefinitionToProtocolVersionMap$22(ConfigRepository.java:672) ~[io.airbyte.airbyte-config-config-persistence-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.db.ExceptionWrappingDatabase.query(ExceptionWrappingDatabase.java:31) ~[io.airbyte.airbyte-db-db-lib-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.config.persistence.ConfigRepository.getActorDefinitionToProtocolVersionMap(ConfigRepository.java:672) ~[io.airbyte.airbyte-config-config-persistence-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.ProtocolVersionChecker.getConflictingActorDefinitions(ProtocolVersionChecker.java:142) ~[io.airbyte-airbyte-bootloader-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.ProtocolVersionChecker.validate(ProtocolVersionChecker.java:96) ~[io.airbyte-airbyte-bootloader-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Bootloader.assertNonBreakingProtocolVersionConstraints(Bootloader.java:157) ~[io.airbyte-airbyte-bootloader-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Bootloader.load(Bootloader.java:106) ~[io.airbyte-airbyte-bootloader-0.50.30.jar:?]
airbyte-bootloader                | 	at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.50.30.jar:?]
airbyte-bootloader                | Caused by: org.postgresql.util.PSQLException: ERROR: relation "public.actor_definition_version" does not exist
airbyte-bootloader                |   Position: 342
airbyte-bootloader                | 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                | 	at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-5.0.1.jar:?]
airbyte-bootloader                | 	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-5.0.1.jar:?]
airbyte-bootloader                | 	at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:219) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.Tools.executeStatementAndGetFirstResultSet(Tools.java:4571) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:236) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:341) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                | 	... 12 more
airbyte-bootloader exited with code 255
service "bootloader" didn't complete successfully: exit 255
2 Views
Open in Slack
PreviousNext
Powered by