Is this your first time deploying Airbyte First on K8s OS Ve

Question

Is this your first time deploying Airbyte First on K8s OS Version Instance EKS Memory Disk 50gb Deployment EKS Airbyte Version 0 40 9 stable with resource limits Hi we re trying to deploy via EKS The Webapp is coming up but the Server and Worker Pods are failing at initialization Server ```2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable AWS ACCESS KEY ID 2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable AWS SECRET ACCESS KEY 2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable SHOULD RUN SYNC WORKFLOWS true 2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable WORKER PLANE CONTROL PLANE 2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable CONFIG DATABASE USER airbyte 2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable CONFIG DATABASE PASSWORD 2022 09 29 00 11 20 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable CONFIG DATABASE URL REMOVED for SLACK 2022 09 29 00 11 21 INFO c z h HikariDataSource lt init gt 80 HikariPool 1 Starting 2022 09 29 00 11 21 INFO c z h HikariDataSource lt init gt 82 HikariPool 1 Start completed 2022 09 29 00 11 21 INFO c z h HikariDataSource lt init gt 80 HikariPool 2 Starting 2022 09 29 00 11 21 INFO c z h HikariDataSource lt init gt 82 HikariPool 2 Start completed 2022 09 29 00 11 22 WARN c n s JsonMetaSchema newValidator 278 Unknown keyword existingJavaType you should define your own Meta Schema If the keyword is irrelevant for valida 2022 09 29 00 11 23 ERROR i a s ServerApp main 336 Server failed java lang IllegalArgumentException null at com google common base Preconditions checkArgument Preconditions java 131 ~ guava 31 0 1 jre jar at io airbyte config storage DefaultS3ClientFactory validateBase DefaultS3ClientFactory java 36 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config storage MinioS3ClientFactory validate MinioS3ClientFactory java 33 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config storage MinioS3ClientFactory lt init gt MinioS3ClientFactory java 27 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config helpers CloudLogs createCloudLogClient CloudLogs java 48 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config helpers LogClientSingleton createCloudClientIfNull LogClientSingleton java 164 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config helpers LogClientSingleton setWorkspaceMdc LogClientSingleton java 151 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte server ServerApp getServer ServerApp java 177 ~ io airbyte airbyte server 0 40 9 jar at io airbyte server ServerApp main ServerApp java 333 ~ io airbyte airbyte server 0 40 9 jar 2022 09 29 00 11 23 INFO c z h HikariDataSource close 350 HikariPool 1 Shutdown initiated 2022 09 29 00 11 23 INFO c z h HikariDataSource close 352 HikariPool 1 Shutdown completed 2022 09 29 00 11 23 INFO c z h HikariDataSource close 350 HikariPool 2 Shutdown initiated 2022 09 29 00 11 23 INFO c z h HikariDataSource close 352 HikariPool 2 Shutdown completed ``` Worker Pod Log ``` | | | | | airbyte workers Micronaut v3 6 3 2022 09 29 00 14 42 INFO i m c e DefaultEnvironment lt init gt 159 Established active environments k8s cloud ec2 control 2022 09 29 00 14 43 INFO c z h HikariDataSource lt init gt 71 HikariPool 1 Starting 2022 09 29 00 14 43 INFO c z h HikariDataSource lt init gt 73 HikariPool 1 Start completed 2022 09 29 00 14 43 INFO c z h HikariDataSource lt init gt 71 HikariPool 2 Starting 2022 09 29 00 14 43 INFO c z h HikariDataSource lt init gt 73 HikariPool 2 Start completed 2022 09 29 00 14 43 INFO i m l PropertiesLoggingLevelsConfigurer configureLogLevelForPrefix 107 Setting log level DEBUG for logger io airbyte bootloader 2022 09 29 00 14 44 INFO i a w c DatabaseBeanFactory configsDatabaseMigrationCheck 139 Configs database configuration removedfromslack 2022 09 29 00 14 44 WARN i a a TrackingClientSingleton get 30 Attempting to fetch an initialized track client Initializing a default one 2022 09 29 00 14 44 INFO i a w t TemporalUtils getTemporalClientWhenConnected 220 Waiting for temporal server 2022 09 29 00 14 44 WARN i a w t TemporalUtils getTemporalClientWhenConnected 231 Waiting for namespace default to be initialized in temporal 2022 09 29 00 14 46 INFO i t s WorkflowServiceStubsImpl lt init gt 188 Created GRPC client for channel ManagedChannelOrphanWrapper delegate=ManagedChannelImpl logId=1 target=airby 2022 09 29 00 14 51 INFO i a w t TemporalUtils getTemporalClientWhenConnected 248 Temporal namespace default initialized 2022 09 29 00 14 51 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable AWS ACCESS KEY ID 2022 09 29 00 14 51 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable AWS SECRET ACCESS KEY 2022 09 29 00 14 51 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable SHOULD RUN SYNC WORKFLOWS true 2022 09 29 00 14 51 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable WORKER PLANE CONTROL PLANE 2022 09 29 00 14 51 WARN i a m l MetricClientFactory getMetricClient 46 MetricClient has not been initialized Must call MetricClientFactory CreateMetricClient before using Metr 2022 09 29 00 14 52 INFO i a w ApplicationInitializer initializeCommonDependencies 174 Initializing common worker dependencies 2022 09 29 00 14 52 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable METRIC CLIENT 2022 09 29 00 14 52 INFO i a c EnvConfigs getEnvOrDefault 1096 Using default value for environment variable METRIC CLIENT 2022 09 29 00 14 52 WARN i a m l MetricClientFactory initialize 74 MetricClient was not recognized or not provided Accepted values are `datadog` or `otel` 2022 09 29 00 14 52 ERROR i m r Micronaut handleStartupException 338 Error starting Micronaut server null java lang IllegalArgumentException null at com google common base Preconditions checkArgument Preconditions java 131 ~ guava 31 1 jre jar at io airbyte config storage DefaultS3ClientFactory validateBase DefaultS3ClientFactory java 36 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config storage MinioS3ClientFactory validate MinioS3ClientFactory java 33 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config storage MinioS3ClientFactory lt init gt MinioS3ClientFactory java 27 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config helpers CloudLogs createCloudLogClient CloudLogs java 48 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config helpers LogClientSingleton createCloudClientIfNull LogClientSingleton java 164 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte config helpers LogClientSingleton setWorkspaceMdc LogClientSingleton java 151 ~ io airbyte airbyte config config models 0 40 9 jar at io airbyte workers ApplicationInitializer initializeCommonDependencies ApplicationInitializer java 180 ~ io airbyte airbyte workers 0 40 9 jar at io airbyte workers ApplicationInitializer onApplicationEvent ApplicationInitializer java 153 ~ io airbyte airbyte workers 0 40 9 jar at io airbyte workers ApplicationInitializer onApplicationEvent ApplicationInitializer java 65 ~ io airbyte airbyte workers 0 40 9 jar at io micronaut context event ApplicationEventPublisherFactory notifyEventListeners ApplicationEventPublisherFactory java 262 ~ micronaut inject 3 6 3 jar 3 6 3 at io micronaut context event ApplicationEventPublisherFactory access$200 ApplicationEventPublisherFactory java 60 ~ micronaut inject 3 6 3 jar 3 6 3 at io micronaut context event ApplicationEventPublisherFactory$2 publishEvent ApplicationEventPublisherFactory java 229 ~ micronaut inject 3 6 3 jar 3 6 3 at io micronaut http server netty NettyHttpServer lambda$fireStartupEvents$15 NettyHttpServer java 574 ~ micronaut http server netty 3 6 3 jar 3 6 3 at java util Optional ifPresent Optional java 178 ~ at io micronaut http server netty NettyHttpServer fireStartupEvents NettyHttpServer java 568 ~ micronaut http server netty 3 6 3 jar 3 6 3 at io micronaut http server netty NettyHttpServer start NettyHttpServer java 297 ~ micronaut http server netty 3 6 3 jar 3 6 3 at io micronaut http server netty NettyHttpServer start NettyHttpServer java 104 ~ micronaut http server netty 3 6 3 jar 3 6 3 at io micronaut runtime Micronaut lambda$start$2 Micronaut java 81 ~ micronaut context 3 6 3 jar 3 6 3 at java util Optional ifPresent Optional java 178 ~ at io micronaut runtime Micronaut start Micronaut java 79 ~ micronaut context 3 6 3 jar 3 6 3 at io micronaut runtime Micronaut run Micronaut java 323 ~ micronaut context 3 6 3 jar 3 6 3 at io micronaut runtime Micronaut run Micronaut java 309 ~ micronaut context 3 6 3 jar 3 6 3 at io airbyte workers Application main Application java 12 ~ io airbyte airbyte workers 0 40 9 jar ```

Dan Siegel · Answer

thanks Nataly

Dan Siegel · Answer

This is our env file ```DATABASE HOST= REMOVED DATABASE DB=airbyte translate manually DATABASE URL=jdbc postgresql $ DATABASE HOST $ DATABASE PORT $ DATABASE DB DATABASE URL= REMOVED Miscellaneous WEBAPP URL=airbyte webapp svc 80 INTERNAL API HOST=airbyte server svc 8001 LOG LEVEL=INFO S3 Minio Log Configuration S3 LOG BUCKET= REMOVED airbyte logs S3 LOG BUCKET REGION= REMOVED State Storage Configuration STATE STORAGE MINIO BUCKET NAME=airbyte dev logs STATE STORAGE MINIO ENDPOINT=<http airbyte minio svc 9000> NORMALIZATION JOB MAIN CONTAINER MEMORY LIMIT=2Gi NORMALIZATION JOB MAIN CONTAINER MEMORY REQUEST=2Gi NORMALIZATION JOB MAIN CONTAINER CPU LIMIT=1 NORMALIZATION JOB MAIN CONTAINER CPU REQUEST=1 CONTAINER ORCHESTRATOR IMAGE=airbyte container orchestrator 0 40 9 MICRONAUT ENVIRONMENTS=control plane Open Telemetry Configuration METRIC CLIENT= OTEL COLLECTOR ENDPOINT=```

Stephen Schlie · Answer

We re using kustomize for the deploy and targetting the `stable with resource limits` then merging the above config in

Dan Siegel · Answer

we re using kustomize

Stephen Schlie · Answer

This is the specific kustomize we re pulling in <https github com airbytehq airbyte tree master kube overlays stable with resource limits>

Dan Siegel · Answer

it ended up being due to us misreading this section here ```Alternatively if you want logs to be sent to a custom location Airbyte currently supports logging to Minio S3 or GCS The following instructions are for users wishing to log to their own Minio layer S3 bucket or GCS bucket ``` we ended up overlooking and leaving the var minio