Dan Siegel
09/29/2022, 12:18 PM2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable SHOULD_RUN_SYNC_WORKFLOWS: 'true'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable WORKER_PLANE: 'CONTROL_PLANE'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable CONFIG_DATABASE_USER: 'airbyte'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '**********'
2022-09-29 00:11:20 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable CONFIG_DATABASE_URL: 'REMOVED for SLACK'
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-09-29 00:11:21 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-09-29 00:11:22 WARN c.n.s.JsonMetaSchema(newValidator):278 - Unknown keyword existingJavaType - you should define your own Meta Schema. If the keyword is irrelevant for valida
2022-09-29 00:11:23 ERROR i.a.s.ServerApp(main):336 - Server failed
java.lang.IllegalArgumentException: null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.0.1-jre.jar:?]
at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.server.ServerApp.getServer(ServerApp.java:177) ~[io.airbyte-airbyte-server-0.40.9.jar:?]
at io.airbyte.server.ServerApp.main(ServerApp.java:333) ~[io.airbyte-airbyte-server-0.40.9.jar:?]
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated...
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):352 - HikariPool-1 - Shutdown completed.
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):350 - HikariPool-2 - Shutdown initiated...
2022-09-29 00:11:23 INFO c.z.h.HikariDataSource(close):352 - HikariPool-2 - Shutdown completed.
Worker Pod Log:
___ _ __ __
/ | (_)____/ /_ __ __/ /____
/ /| | / / ___/ __ \/ / / / __/ _ \
/ ___ |/ / / / /_/ / /_/ / /_/ __/
/_/ |_/_/_/ /_.___/\__, /\__/\___/
/____/
: airbyte-workers :
Micronaut (v3.6.3)
2022-09-29 00:14:42 INFO i.m.c.e.DefaultEnvironment(<init>):159 - Established active environments: [k8s, cloud, ec2, control]
2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-1 - Starting...
2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-1 - Start completed.
2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):71 - HikariPool-2 - Starting...
2022-09-29 00:14:43 INFO c.z.h.HikariDataSource(<init>):73 - HikariPool-2 - Start completed.
2022-09-29 00:14:43 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):107 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2022-09-29 00:14:44 INFO i.a.w.c.DatabaseBeanFactory(configsDatabaseMigrationCheck):139 - Configs database configuration: removedfromslack
2022-09-29 00:14:44 WARN i.a.a.TrackingClientSingleton(get):30 - Attempting to fetch an initialized track client. Initializing a default one.
2022-09-29 00:14:44 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):220 - Waiting for temporal server...
2022-09-29 00:14:44 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):231 - Waiting for namespace default to be initialized in temporal...
2022-09-29 00:14:46 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airby
2022-09-29 00:14:51 INFO i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):248 - Temporal namespace default initialized!
2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_ACCESS_KEY_ID: ''
2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable AWS_SECRET_ACCESS_KEY: ''
2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable SHOULD_RUN_SYNC_WORKFLOWS: 'true'
2022-09-29 00:14:51 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable WORKER_PLANE: 'CONTROL_PLANE'
2022-09-29 00:14:51 WARN i.a.m.l.MetricClientFactory(getMetricClient):46 - MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using Metr
2022-09-29 00:14:52 INFO i.a.w.ApplicationInitializer(initializeCommonDependencies):174 - Initializing common worker dependencies.
2022-09-29 00:14:52 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: ''
2022-09-29 00:14:52 INFO i.a.c.EnvConfigs(getEnvOrDefault):1096 - Using default value for environment variable METRIC_CLIENT: ''
2022-09-29 00:14:52 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`.
2022-09-29 00:14:52 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
at io.airbyte.config.storage.DefaultS3ClientFactory.validateBase(DefaultS3ClientFactory.java:36) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.storage.MinioS3ClientFactory.validate(MinioS3ClientFactory.java:33) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.storage.MinioS3ClientFactory.<init>(MinioS3ClientFactory.java:27) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.helpers.CloudLogs.createCloudLogClient(CloudLogs.java:48) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.helpers.LogClientSingleton.createCloudClientIfNull(LogClientSingleton.java:164) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.config.helpers.LogClientSingleton.setWorkspaceMdc(LogClientSingleton.java:151) ~[io.airbyte.airbyte-config-config-models-0.40.9.jar:?]
at io.airbyte.workers.ApplicationInitializer.initializeCommonDependencies(ApplicationInitializer.java:180) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:153) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
at io.airbyte.workers.ApplicationInitializer.onApplicationEvent(ApplicationInitializer.java:65) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:262) ~[micronaut-inject-3.6.3.jar:3.6.3]
at io.micronaut.context.event.ApplicationEventPublisherFactory.access$200(ApplicationEventPublisherFactory.java:60) ~[micronaut-inject-3.6.3.jar:3.6.3]
at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:229) ~[micronaut-inject-3.6.3.jar:3.6.3]
at io.micronaut.http.server.netty.NettyHttpServer.lambda$fireStartupEvents$15(NettyHttpServer.java:574) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:568) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:297) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:104) ~[micronaut-http-server-netty-3.6.3.jar:3.6.3]
at io.micronaut.runtime.Micronaut.lambda$start$2(Micronaut.java:81) ~[micronaut-context-3.6.3.jar:3.6.3]
at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
at io.micronaut.runtime.Micronaut.start(Micronaut.java:79) ~[micronaut-context-3.6.3.jar:3.6.3]
at io.micronaut.runtime.Micronaut.run(Micronaut.java:323) ~[micronaut-context-3.6.3.jar:3.6.3]
at io.micronaut.runtime.Micronaut.run(Micronaut.java:309) ~[micronaut-context-3.6.3.jar:3.6.3]
at io.airbyte.workers.Application.main(Application.java:12) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
DATABASE_HOST=**REMOVED**
DATABASE_DB=airbyte
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_DB}
DATABASE_URL=**REMOVED**
# Miscellaneous
WEBAPP_URL=airbyte-webapp-svc:80
INTERNAL_API_HOST=airbyte-server-svc:8001
LOG_LEVEL=INFO
# S3/Minio Log Configuration
S3_LOG_BUCKET=**REMOVED**-airbyte-logs
S3_LOG_BUCKET_REGION=**REMOVED**
# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=2Gi
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=2Gi
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_LIMIT=1
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_REQUEST=1
CONTAINER_ORCHESTRATOR_IMAGE=airbyte/container-orchestrator:0.40.9
MICRONAUT_ENVIRONMENTS=control-plane
# Open Telemetry Configuration
METRIC_CLIENT=
OTEL_COLLECTOR_ENDPOINT=
Stephen Schlie
09/29/2022, 4:51 PMstable-with-resource-limits
then merging the above config inDan Siegel
09/29/2022, 6:23 PMStephen Schlie
09/29/2022, 6:34 PMDan Siegel
10/05/2022, 9:45 PMAlternatively, if you want logs to be sent to a custom location, Airbyte currently supports logging to Minio, S3 or GCS. The following instructions are for users wishing to log to their own Minio layer, S3 bucket or GCS bucket.
we ended up overlooking and leaving the var minio