https://linen.dev logo
Join Slack
Powered by
Hi team, I'm configuring postgres db as data sourc...
# ask-community-for-troubleshooting
m
Hi team, I'm configuring postgres db as data source which is deployed in EC2 instance. My airbyte is deployed as k8s pod in 
dev
namespace in EKS cluster. Can anyone let me know what's the issue..
👀 1
✅ 1
h
Hey can you share the logs of server/scheduler?
m
@Harshith (Airbyte) These are airbyte scheduler logs
Copy code
2022-03-07 08:16:51 INFO i.a.s.a.SchedulerApp(main):227 - workspaceRoot = /workspace
2022-03-07 08:16:51 INFO i.a.s.a.SchedulerApp(main):230 - temporalHost = airbyte-temporal-svc:7233
2022-03-07 08:16:53 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:16:57 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:01 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
Cannot start publish with com.van.logging.aws.S3PublishHelper@4d9a9624 due to error: Cannot start publishing: Unable to execute HTTP request: Connect to airbyte-minio-svc:9000 [airbyte-minio-svc/172.20.232.12] failed: Connect timed out
Cannot end publish with com.van.logging.aws.S3PublishHelper@4d9a9624 due to error: Cannot end publishing: Cannot publish to S3: The Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 16DA0B49B7D4B4DF; S3 Extended Request ID: 854e0863-7c06-4a5f-b7ed-25865848bb2e; Proxy: null)
2022-03-07 08:17:05 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:07 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:09 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:11 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:13 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:15 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:17 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:19 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:21 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:23 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:25 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:27 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:29 INFO i.a.s.a.SchedulerApp(waitForServer):213 - Waiting for server to become available...
2022-03-07 08:17:32 INFO i.a.s.a.SchedulerApp(main):235 - Creating Job DB connection pool...
2022-03-07 08:17:32 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-07 08:17:32 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2022-03-07 08:17:32 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-07 08:17:32 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_USER: 'postgres_user'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev>'
2022-03-07 08:17:32 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-07 08:17:32 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):28 - Testing if airbyte_configs has been created and seeded...
2022-03-07 08:17:32 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-07 08:17:32 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable MINIMUM_WORKSPACE_RETENTION_DAYS: '1'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable MAXIMUM_WORKSPACE_RETENTION_DAYS: '60'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable MAXIMUM_WORKSPACE_SIZE_MB: '5000'
2022-03-07 08:17:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-03-07 08:17:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '10'
2022-03-07 08:17:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable ACTIVITY_MAX_TIMEOUT_SECOND: '30'
2022-03-07 08:17:33 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-03-07 08:17:33 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-03-07 08:17:35 INFO i.t.s.WorkflowServiceStubsImpl(<init>):184 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-03-07 08:17:40 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable PUBLISH_METRICS: 'false'
2022-03-07 08:17:40 INFO i.a.s.a.SchedulerApp(main):278 - Launching scheduler...
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SYNC_JOB_MAX_ATTEMPTS: '3'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_REQUEST: 'null'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_CPU_LIMIT: 'null'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_REQUEST: 'null'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_MAIN_CONTAINER_MEMORY_LIMIT: 'null'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_TOLERATIONS: ''
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_NODE_SELECTORS: ''
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET: ''
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_SOCAT_IMAGE: 'alpine/socat:1.7.4.1-r1'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_BUSYBOX_IMAGE: 'busybox:1.28'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_CURL_IMAGE: 'curlimages/curl:7.77.0'
2022-03-07 08:17:40 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable DEFAULT_WORKER_STATUS_CHECK_INTERVAL: 'PT30S'
2022-03-07 08:17:40 INFO i.a.c.f.EnvVariableFeatureFlags(usesNewScheduler):14 - New Scheduler: false
2022-03-07 08:17:40 INFO i.a.s.a.SchedulerApp(start):146 - Start running the old scheduler
Cannot end publish with com.van.logging.aws.S3PublishHelper@4d9a9624 due to error: Cannot end publishing: Cannot publish to S3: The Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 16DA0B54945399B2; S3 Extended Request ID: 854e0863-7c06-4a5f-b7ed-25865848bb2e; Proxy: null)
@Harshith (Airbyte) here are the server logs
Copy code
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_TOLERATIONS: ''
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_NODE_SELECTORS: ''
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET: ''
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY: 'IfNotPresent'
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_SOCAT_IMAGE: 'alpine/socat:1.7.4.1-r1'
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_BUSYBOX_IMAGE: 'busybox:1.28'
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOB_KUBE_CURL_IMAGE: 'curlimages/curl:7.77.0'
2022-03-08 16:58:32 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable DEFAULT_WORKER_STATUS_CHECK_INTERVAL: 'PT30S'
2022-03-08 16:58:32 INFO i.a.s.ServerApp(getServer):151 - Checking databases..
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_USER: 'postgres_user'
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev>'
2022-03-08 16:58:33 INFO i.a.s.ServerApp(assertDatabasesReady):128 - Checking configs database flyway migration version..
2022-03-08 16:58:33 INFO i.a.d.i.MinimumFlywayMigrationVersionCheck(assertDatabase):42 - Waiting for database...
2022-03-08 16:58:33 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-08 16:58:33 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$5):127 - Testing airbyte configs database connection...
Cannot end publish with com.van.logging.aws.S3PublishHelper@68130a64 due to error: Cannot end publishing: Cannot publish to S3: The Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 16DA765320177B3A; S3 Extended Request ID: c1ac9810-2e0a-4339-82db-4faccc34ebe0; Proxy: null)
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-08 16:58:33 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$isInitialized$4):62 - The airbyte configs database is initialized
2022-03-08 16:58:33 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-08 16:58:33 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):28 - Testing if airbyte_configs has been created and seeded...
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-03-08 16:58:33 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-03-08 16:58:33 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev> (PostgreSQL 14.1)
2022-03-08 16:58:33 WARN o.f.c.i.l.s.Slf4jLog(warn):53 - Flyway upgrade recommended: PostgreSQL 14.1 is newer than this version of Flyway and support has not been tested. The latest supported version of PostgreSQL is 13.
2022-03-08 16:58:33 INFO i.a.d.i.MinimumFlywayMigrationVersionCheck(assertMigrations):76 - Current database migration version 0.35.28.001
2022-03-08 16:58:33 INFO i.a.d.i.MinimumFlywayMigrationVersionCheck(assertMigrations):77 - Minimum Flyway version required 0.35.15.001
2022-03-08 16:58:33 INFO i.a.s.ServerApp(assertDatabasesReady):134 - Checking jobs database flyway migration version..
2022-03-08 16:58:33 INFO i.a.d.i.MinimumFlywayMigrationVersionCheck(assertDatabase):42 - Waiting for database...
2022-03-08 16:58:33 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-08 16:58:33 INFO i.a.d.i.BaseDatabaseInstance(lambda$isDatabaseConnected$5):127 - Testing airbyte jobs database connection...
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-08 16:58:33 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-08 16:58:33 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-03-08 16:58:33 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-03-08 16:58:33 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev> (PostgreSQL 14.1)
2022-03-08 16:58:33 WARN o.f.c.i.l.s.Slf4jLog(warn):53 - Flyway upgrade recommended: PostgreSQL 14.1 is newer than this version of Flyway and support has not been tested. The latest supported version of PostgreSQL is 13.
2022-03-08 16:58:33 INFO i.a.d.i.MinimumFlywayMigrationVersionCheck(assertMigrations):76 - Current database migration version 0.35.5.001
2022-03-08 16:58:33 INFO i.a.d.i.MinimumFlywayMigrationVersionCheck(assertMigrations):77 - Minimum Flyway version required 0.29.15.001
2022-03-08 16:58:33 INFO i.a.s.ServerApp(getServer):158 - Creating Staged Resource folder...
2022-03-08 16:58:33 INFO i.a.s.ServerApp(getServer):161 - Creating config repository...
2022-03-08 16:58:33 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-08 16:58:33 INFO i.a.d.i.c.ConfigsDatabaseInstance(lambda$static$3):28 - Testing if airbyte_configs has been created and seeded...
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable SECRET_PERSISTENCE: 'NONE'
2022-03-08 16:58:33 INFO i.a.s.ServerApp(getServer):170 - Creating jobs persistence...
2022-03-08 16:58:33 WARN i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):66 - Waiting for database to become available...
2022-03-08 16:58:33 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):25 - Testing if jobs database is ready...
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetryTimeout):91 - Database available!
2022-03-08 16:58:33 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):49 - Database available!
2022-03-08 16:58:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable DEPLOYMENT_MODE: 'OSS'
2022-03-08 16:58:34 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable ACTIVITY_MAX_ATTEMPT: '10'
2022-03-08 16:58:34 INFO i.a.c.EnvConfigs(getEnvOrDefault):763 - Using default value for environment variable ACTIVITY_MAX_TIMEOUT_SECOND: '30'
2022-03-08 16:58:34 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-03-08 16:58:34 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-03-08 16:58:36 INFO i.t.s.WorkflowServiceStubsImpl(<init>):184 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte-temporal-svc:7233}}
2022-03-08 16:58:41 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-03-08 16:58:41 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-03-08 16:58:41 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-03-08 16:58:43 INFO i.t.s.WorkflowServiceStubsImpl(<init>):184 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=5, target=airbyte-temporal-svc:7233}}
2022-03-08 16:58:48 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-03-08 16:58:48 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for temporal server...
2022-03-08 16:58:48 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):185 - Waiting for default namespace to be initialized in temporal...
2022-03-08 16:58:50 INFO i.t.s.WorkflowServiceStubsImpl(<init>):184 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=9, target=airbyte-temporal-svc:7233}}
2022-03-08 16:58:55 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):201 - Found temporal default namespace!
2022-03-08 16:58:55 INFO i.a.s.ServerApp(getServer):200 - Starting server...
2022-03-08 16:58:55 INFO o.e.j.u.l.Log(initialized):169 - Logging initialized @23710ms to org.eclipse.jetty.util.log.Slf4jLog
2022-03-08 16:58:55 INFO o.e.j.s.Server(doStart):360 - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 17.0.1+12-39
Mar 08, 2022 4:58:55 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2022-03-08 16:58:55 INFO o.h.v.i.u.Version(<clinit>):21 - HV000001: Hibernate Validator 6.1.2.Final
2022-03-08 16:58:56 INFO o.e.j.s.h.ContextHandler(doStart):860 - Started o.e.j.s.ServletContextHandler@46d51d5e{/,null,AVAILABLE}
2022-03-08 16:58:56 INFO o.e.j.s.AbstractConnector(doStart):331 - Started ServerConnector@24a3d414{HTTP/1.1, (http/1.1)}{0.0.0.0:8001}
2022-03-08 16:58:56 INFO o.e.j.s.Server(doStart):400 - Started @24282ms
2022-03-08 16:58:56 INFO i.a.s.ServerApp(start):120 - 

--------------------------------------
 Now ready at <http://localhost:8000/>
--------------------------------------
Version: 0.35.31-alpha

2022-03-08 16:58:58 INFO i.a.s.RequestLogger(filter):95 - REQ 10.0.2.253 GET 200 /api/v1/health
Cannot end publish with com.van.logging.aws.S3PublishHelper@68130a64 due to error: Cannot end publishing: Cannot publish to S3: The Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 16DA766112DB48C3; S3 Extended Request ID: c1ac9810-2e0a-4339-82db-4faccc34ebe0; Proxy: null)
All pods in the namespace are up & running
This is my .env file
Copy code
AIRBYTE_VERSION=0.35.31-alpha

# Airbyte Internal Database, see <https://docs.airbyte.io/operator-guides/configuring-airbyte-db>
DATABASE_HOST=<http://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com|cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com>
DATABASE_PORT=5432
DATABASE_DB=airbyte_dev
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT/${DATABASE_DB}
DATABASE_URL=jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev>
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001
CONFIG_DATABASE_URL=jdbc:<postgresql://cdpass-dev-postgres-instance.cxlcw2mudakk.us-east-2.rds.amazonaws.com:5432/airbyte_dev>

# When using the airbyte-db via default docker image:
CONFIG_ROOT=/configs
DATA_DOCKER_MOUNT=airbyte_data
DB_DOCKER_MOUNT=airbyte_db

RUN_DATABASE_MIGRATION_ON_STARTUP=true

# <http://Temporal.io|Temporal.io> worker configuration
TEMPORAL_HOST=airbyte-temporal-svc:7233
TEMPORAL_WORKER_PORTS=9001,9002,9003,9004,9005,9006,9007,9008,9009,9010,9011,9012,9013,9014,9015,9016,9017,9018,9019,9020,9021,9022,9023,9024,9025,9026,9027,9028,9029,9030,9031,9032,9033,9034,9035,9036,9037,9038,9039,9040

# Workspace storage for running jobs (logs, etc)
WORKSPACE_ROOT=/workspace
WORKSPACE_DOCKER_MOUNT=airbyte_workspace

LOCAL_ROOT=/tmp/airbyte_local

# Maximum total simultaneous jobs across all worker nodes
SUBMITTER_NUM_THREADS=10

# Miscellaneous
TRACKING_STRATEGY=segment
WEBAPP_URL=airbyte-webapp-svc:80
API_URL=/api/v1/
INTERNAL_API_HOST=airbyte-server-svc:8001

WORKER_ENVIRONMENT=kubernetes
FULLSTORY=enabled
IS_DEMO=false
LOG_LEVEL=INFO

# S3/Minio Log Configuration
S3_LOG_BUS3_LOG_BUCKET_REGION=
CKET=airbyte-dev-logs
S3_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>
S3_PATH_STYLE_ACCESS=true

# GCS Log Configuration
GCS_LOG_BUCKET=

# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>

# Docker Resource Limits
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=

# Worker pod tolerations and node selectors
JOB_KUBE_TOLERATIONS=
JOB_KUBE_NODE_SELECTORS=

# Job image pull policy
JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY=

# Launch a separate pod to orchestrate sync steps
CONTAINER_ORCHESTRATOR_ENABLED=false
m
m
@Marcos Marx (Airbyte) I've added the following environment variables in 
.env
file and it worked. Before this I had 
AWS_ACCESS_KEY
and 
SECRET_KEY
were in 
secrets
file.
Copy code
S3_LOG_BUCKET=<your_s3_bucket_to_write_logs_in>
S3_LOG_BUCKET_REGION=<your_s3_bucket_region>
AWS_ACCESS_KEY_ID=<your_aws_access_key_id>
AWS_SECRET_ACCESS_KEY=<your_aws_secret_access_key>
# Set this to empty.
S3_MINIO_ENDPOINT=
# Set this to empty.
S3_PATH_STYLE_ACCESS=
Thanks for your help.
3 Views
Open in Slack
PreviousNext
Powered by