Bob Seehra
01/25/2023, 9:43 PM2023-01-25 01:50:54 [32mINFO[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
The sync fails when we have the replication-orchestrator enabled. Disabling this allows the syncs to work, although we still see the exception in the logs (see thread for full stack trace)
CONTAINER_ORCHESTRATOR_ENABLED=true
Are there additional environment settings we are missing that need to be added?Bob Seehra
01/25/2023, 9:44 PMjava.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:116) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.29.jar:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:234) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
... 3 more
Caused by: io.airbyte.workers.exception.WorkerException: Failed to create pod orchestrator-repl-job-81006-attempt-0, pre-existing pod exists which didn't advance out of the NOT_STARTED state.
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:198) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
... 3 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.70.128.1/api/v1/namespaces/airbyte-dev/pods>. Message: Pod "orchestrator-repl-job-81006-attempt-0" is invalid: [spec.volumes[1].secret.secretName: Required value, spec.containers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.containers[0].volumeMounts[1].mountPath: Required value, spec.initContainers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.initContainers[0].volumeMounts[1].mountPath: Required value]. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.volumes[1].secret.secretName, message=Required value, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec.containers[0].volumeMounts[1].name, message=Not found: "airbyte-secret", reason=FieldValueNotFound, additionalProperties={}), StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, message=Required value, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec.initContainers[0].volumeMounts[1].name, message=Not found: "airbyte-secret", reason=FieldValueNotFound, additionalProperties={}), StatusCause(field=spec.initContainers[0].volumeMounts[1].mountPath, message=Required value, reason=FieldValueRequired, additionalProperties={})], group=null, kind=Pod, name=orchestrator-repl-job-81006-attempt-0, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "orchestrator-repl-job-81006-attempt-0" is invalid: [spec.volumes[1].secret.secretName: Required value, spec.containers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.containers[0].volumeMounts[1].mountPath: Required value, spec.initContainers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.initContainers[0].volumeMounts[1].mountPath: Required value], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:555) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:305) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:48) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:318) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:308) ~[kubernetes-client-5.12.2.jar:?]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.create(AsyncOrchestratorPodProcess.java:438) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:190) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
... 3 morePaul Grolleau
01/24/2024, 1:04 PMSebastien HO
01/25/2024, 9:29 AMSebastien HO
01/25/2024, 9:29 AMPaul Grolleau
01/25/2024, 9:41 AM- name: "CONTAINER_ORCHESTRATOR_SECRET_NAME"
value: "gcs-log-creds"
- name: "CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH"
value: "/secrets/gcs-log-creds-worker"Sebastien HO
01/25/2024, 10:17 AMCONTAINER_ORCHESTRATOR_SECRET_NAME : it should contains the secret to connect to gcp
⢠CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH : path where to mount the secret gcp on local orchestrator variable -> it should reflect the GCP