https://linen.dev logo
Join Slack
Powered by
Hey Airbyte team! Is it possible to use different ...
# replication-troubleshooting
m
Hey Airbyte team! Is it possible to use different schema for Airbyte Postgres database?
I am getting this error in airbyte-bootloader service:
Copy code
2023-01-11 17:07:52 INFO i.a.b.BootloaderApp(load):174 - Ran Flyway migrations.
Exception in thread "main" org.jooq.exception.DataAccessException: SQL [select * from "public"."workspace"]; ERROR: relation "public.workspace" does not exist
  Position: 15
        at org.jooq_3.13.4.POSTGRES.debug(Unknown Source)
        at org.jooq.impl.Tools.translate(Tools.java:2753)
        at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:755)
        at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:385)
        at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:354)
        at org.jooq.impl.SelectImpl.fetch(SelectImpl.java:2690)
        at io.airbyte.config.persistence.DatabaseConfigPersistence.lambda$listStandardWorkspaceWithMetadata$15(DatabaseConfigPersistence.java:345)
        at io.airbyte.db.Database.query(Database.java:23)
        at io.airbyte.db.ExceptionWrappingDatabase.query(ExceptionWrappingDatabase.java:23)
        at io.airbyte.config.persistence.DatabaseConfigPersistence.listStandardWorkspaceWithMetadata(DatabaseConfigPersistence.java:340)
        at io.airbyte.config.persistence.DatabaseConfigPersistence.listStandardWorkspaceWithMetadata(DatabaseConfigPersistence.java:336)
        at io.airbyte.config.persistence.DatabaseConfigPersistence.listConfigsWithMetadata(DatabaseConfigPersistence.java:303)
        at io.airbyte.config.persistence.DatabaseConfigPersistence.listConfigs(DatabaseConfigPersistence.java:260)
        at io.airbyte.config.persistence.ValidatingConfigPersistence.listConfigs(ValidatingConfigPersistence.java:49)
        at io.airbyte.config.persistence.ConfigRepository.listStandardWorkspaces(ConfigRepository.java:153)
        at io.airbyte.bootloader.BootloaderApp.createWorkspaceIfNoneExists(BootloaderApp.java:276)
        at io.airbyte.bootloader.BootloaderApp.load(BootloaderApp.java:179)
        at io.airbyte.bootloader.BootloaderApp.main(BootloaderApp.java:260)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "public.workspace" does not exist
  Position: 15
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:167)
        at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:156)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
        at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:209)
        at org.jooq.impl.Tools.executeStatementAndGetFirstResultSet(Tools.java:3992)
        at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:295)
        at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:371)
        ... 14 more
s
Hey, how are you deploying Airbyte? The airbyte db should be configured on startup.
m
@Sunny Hashmi I have configured Airbyte to a AWS RDS Postgres for internal DB, but when bootloader starts it gives this issue. Could you help me resolve this issue?
Copy code
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(initializeDatabases):207 - Databases initialized.
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(load):104 - Checking migration compatibility...
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(assertNonBreakingMigration):136 - Checking for illegal upgrade...
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(getRequiredVersionUpgrade):214 - No previous Airbyte Version set.
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(load):107 - Checking protocol version constraints...
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.ProtocolVersionChecker(validate):83 - No previous version of Airbyte detected, assuming this is a fresh deploy.
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(trackProtocolVersion):254 - AirbyteProtocol version support range: [0.0.0:0.3.0]
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(load):110 - Running database migrations...
airbyte-bootloader                | 2023-11-16 11:29:00 INFO i.a.b.Bootloader(runFlywayMigration):235 - Creating baseline for config database...
airbyte-bootloader                | 2023-11-16 11:29:00 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
airbyte-bootloader                | 2023-11-16 11:29:00 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:<postgresql://aurora-postgres.cluster-cgbzclqbwtia.ap-southeast-1.rds.amazonaws.com:5432/airbyte> (PostgreSQL 15.3)
airbyte-bootloader                | 2023-11-16 11:29:01 WARN o.f.c.i.l.s.Slf4jLog(warn):53 - Flyway upgrade recommended: PostgreSQL 15.3 is newer than this version of Flyway and support has not been tested. The latest supported version of PostgreSQL is 13.
airbyte-bootloader                | 2023-11-16 11:29:01 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Creating Schema History table "airbyte"."airbyte_configs_migrations" with baseline ...
airbyte-bootloader                | 2023-11-16 11:29:01 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully baselined schema with version: 0.29.0.001
airbyte-bootloader                | 2023-11-16 11:29:01 INFO i.a.b.Bootloader(runFlywayMigration):237 - Creating baseline for job database...
airbyte-bootloader                | 2023-11-16 11:29:01 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
airbyte-bootloader                | 2023-11-16 11:29:01 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:<postgresql://aurora-postgres.cluster-cgbzclqbwtia.ap-southeast-1.rds.amazonaws.com:5432/airbyte> (PostgreSQL 15.3)
airbyte-bootloader                | 2023-11-16 11:29:01 WARN o.f.c.i.l.s.Slf4jLog(warn):53 - Flyway upgrade recommended: PostgreSQL 15.3 is newer than this version of Flyway and support has not been tested. The latest supported version of PostgreSQL is 13.
airbyte-bootloader                | 2023-11-16 11:29:01 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Creating Schema History table "airbyte"."airbyte_jobs_migrations" with baseline ...
airbyte-bootloader                | 2023-11-16 11:29:01 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully baselined schema with version: 0.29.0.001
airbyte-bootloader                | 2023-11-16 11:29:01 INFO i.a.b.Bootloader(runFlywayMigration):246 - Auto database migration has been skipped.
airbyte-bootloader                | 2023-11-16 11:29:01 INFO i.a.b.Bootloader(load):113 - Creating workspace (if none exists)...
airbyte-bootloader                | 2023-11-16 11:29:01 ERROR i.a.b.Application(main):25 - Unable to bootstrap Airbyte environment.
airbyte-bootloader                | org.jooq.exception.DataAccessException: SQL [select "public"."workspace".* from "public"."workspace"]; ERROR: relation "public.workspace" does not exist
airbyte-bootloader                |   Position: 36
airbyte-bootloader                |     at org.jooq_3.17.8.POSTGRES.debug(Unknown Source) ~[?:?]
airbyte-bootloader                |     at org.jooq.impl.Tools.translate(Tools.java:3313) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.DefaultExecuteContext.sqlException(DefaultExecuteContext.java:678) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:355) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractResultQuery.fetch(AbstractResultQuery.java:290) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.SelectImpl.fetch(SelectImpl.java:2837) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at io.airbyte.data.services.impls.jooq.WorkspaceServiceJooqImpl.lambda$listWorkspaceQuery$5(WorkspaceServiceJooqImpl.java:172) ~[io.airbyte-airbyte-data-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.db.Database.query(Database.java:23) ~[io.airbyte.airbyte-db-db-lib-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.db.ExceptionWrappingDatabase.query(ExceptionWrappingDatabase.java:31) ~[io.airbyte.airbyte-db-db-lib-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.data.services.impls.jooq.WorkspaceServiceJooqImpl.listWorkspaceQuery(WorkspaceServiceJooqImpl.java:168) ~[io.airbyte-airbyte-data-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.data.services.impls.jooq.WorkspaceServiceJooqImpl.listStandardWorkspaces(WorkspaceServiceJooqImpl.java:142) ~[io.airbyte-airbyte-data-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.config.persistence.ConfigRepository.listStandardWorkspaces(ConfigRepository.java:309) ~[io.airbyte.airbyte-config-config-persistence-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.bootloader.Bootloader.createWorkspaceIfNoneExists(Bootloader.java:180) ~[io.airbyte-airbyte-bootloader-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.bootloader.Bootloader.load(Bootloader.java:114) ~[io.airbyte-airbyte-bootloader-0.50.34.jar:?]
airbyte-bootloader                |     at io.airbyte.bootloader.Application.main(Application.java:22) ~[io.airbyte-airbyte-bootloader-0.50.34.jar:?]
airbyte-bootloader                | Caused by: org.postgresql.util.PSQLException: ERROR: relation "public.workspace" does not exist
airbyte-bootloader                |   Position: 36
airbyte-bootloader                |     at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177) ~[postgresql-42.5.4.jar:42.5.4]
airbyte-bootloader                |     at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44) ~[HikariCP-5.0.1.jar:?]
airbyte-bootloader                |     at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java) ~[HikariCP-5.0.1.jar:?]
airbyte-bootloader                |     at org.jooq.tools.jdbc.DefaultPreparedStatement.execute(DefaultPreparedStatement.java:219) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.Tools.executeStatementAndGetFirstResultSet(Tools.java:4571) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractResultQuery.execute(AbstractResultQuery.java:236) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     at org.jooq.impl.AbstractQuery.execute(AbstractQuery.java:341) ~[jooq-3.17.8.jar:?]
airbyte-bootloader                |     ... 11 more
airbyte-bootloader exited with code 255
service "bootloader" didn't complete successfully: exit 255
I am doing the airbyte deployment on EC2 using latest Docker image
@[DEPRECATED] Marcos Marx can you offer any inputs here?
@Marcos Marx (Airbyte)
cc: @Claudio Reggiani
7 Views
Open in Slack
PreviousNext
Powered by