Title
d

Dan Siegel

09/29/2022, 12:18 PM
• Is this your first time deploying Airbyte?: First on K8s • OS Version / Instance: EKS • Memory / Disk: 50gb • Deployment: EKS • Airbyte Version: 0.40.9 - stable with resource limits Hi - we're trying to deploy via EKS. The Webapp is coming up, but the Server and Worker Pods are failing at initialization: Server:
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable SHOULD_RUN_SYNC_WORKFLOWS: 'true'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable WORKER_PLANE: 'CONTROL_PLANE'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable CONFIG_DATABASE_USER: 'airbyte'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '**********'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable CONFIG_DATABASE_URL: 'REMOVED for SLACK'
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-09-29 00:11:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for valida
2022-09-29 00:11:23 ERROR i.a.s.ServerApp(main):336 - Server failed
java.lang.IllegalArgumentException: null
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.0.1-jre.jar:?]
    at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
    at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
    at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
    at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
    at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
    at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
    at io.airbyte.server.ServerApp.getServer(ServerApp.java:177) ~[io.airbyte-airbyte-server-0.40.9.jar:?]
    at io.airbyte.server.ServerApp.main(ServerApp.java:333) ~[io.airbyte-airbyte-server-0.40.9.jar:?]
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated...
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):352 - HikariPool-1 - Shutdown completed.
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):350 - HikariPool-2 - Shutdown initiated...
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):352 - HikariPool-2 - Shutdown completed.
Worker Pod Log:
___    _      __          __
    /   |  (_)____/ /_  __  __/ /____
   / /| | / / ___/ __ \/ / / / __/ _ \
  / ___ |/ / /  / /_/ / /_/ / /_/  __/
 /_/  |_/_/_/  /_.___/\__, /\__/\___/
                     /____/
         : airbyte-workers :
 
   Micronaut (v3.6.3)
 
 2022-09-29 00:14:42 INFO i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2, control]
 2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
 2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-1 - Start completed.
 2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-2 - Starting...
 2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-2 - Start completed.
 2022-09-29 00:14:43 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
 2022-09-29 00:14:44 INFO i.a.w.c.DatabaseBeanFactory(configsDatabaseMigrationCheck):139 - Configs database configuration: removedfromslack
 2022-09-29 00:14:44 WARN i.a.a.TrackingClientSingleton(get):30 - Attempting to fetch an initialized track client. Initializing a default one.
 2022-09-29 00:14:44 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):220 - Waiting for temporal server...
 2022-09-29 00:14:44 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):231 - Waiting for namespace default to be initialized in temporal...
 2022-09-29 00:14:46 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airby
 2022-09-29 00:14:51 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):248 - Temporal namespace default initialized!
 2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
 2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
 2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable SHOULD_RUN_SYNC_WORKFLOWS: 'true'
 2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable WORKER_PLANE: 'CONTROL_PLANE'
 2022-09-29 00:14:51 WARN i.a.m.l.MetricClientFactory(getMetricClient):46 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using Metr
 2022-09-29 00:14:52 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):174 - Initializing common worker dependencies.
 2022-09-29 00:14:52 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: ''
 2022-09-29 00:14:52 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: ''
 2022-09-29 00:14:52 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`.
 2022-09-29 00:14:52 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
 java.lang.IllegalArgumentException: null
     at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
     at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
     at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
     at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
     at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
     at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
     at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
     at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:180) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
     at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:153) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
     at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:65) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
     at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.6.3.jar:3.6.3]
     at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.6.3.jar:3.6.3]
     at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.6.3.jar:3.6.3]
     at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:574) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
     at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
     at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:568) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
     at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:297) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
     at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
     at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.6.3.jar:3.6.3]
     at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
     at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.6.3.jar:3.6.3]
     at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.6.3.jar:3.6.3]
     at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.6.3.jar:3.6.3]
     at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
thanks Nataly
This is our .env file
DATABASE_HOST=**REMOVED**
DATABASE_DB=airbyte
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB}
DATABASE_URL=**REMOVED**

# Miscellaneous
WEBAPP_URL=airbyte-webapp-svc:80
INTERNAL_API_HOST=airbyte-server-svc:8001

LOG_LEVEL=INFO

# S3/Minio Log Configuration
S3_LOG_BUCKET=**REMOVED**-airbyte-logs
S3_LOG_BUCKET_REGION=**REMOVED**

# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>

NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=2Gi
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=2Gi
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_LIMIT=1
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_REQUEST=1

CONTAINER_ORCHESTRATOR_IMAGE=airbyte/container-orchestrator:0.40.9
MICRONAUT_ENVIRONMENTS=control-plane

# Open Telemetry Configuration
METRIC_CLIENT=
OTEL_COLLECTOR_ENDPOINT=
s

Stephen Schlie

09/29/2022, 4:51 PM
We're using kustomize for the deploy and targetting the
stable-with-resource-limits
then merging the above config in
d

Dan Siegel

09/29/2022, 6:23 PM
we're using kustomize
s

Stephen Schlie

09/29/2022, 6:34 PM
d

Dan Siegel

10/05/2022, 9:45 PM
it ended up being due to us misreading this section here:
Alternatively, if you want logs to be sent to a custom location, Airbyte currently supports logging to Minio, S3 or GCS. The following instructions are for users wishing to log to their own Minio layer, S3 bucket or GCS bucket.
we ended up overlooking and leaving the var minio