Hi All! We are upgrading from 0.40.4 to 0.40.29 in...
# replication-troubleshooting
b
Hi All! We are upgrading from 0.40.4 to 0.40.29 in our test environment running on GKE and are now experiencing issues with the replication-orchestrator for all of our syncs.
Copy code
2023-01-25 01:50:54 [32mINFO[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
The sync fails when we have the replication-orchestrator enabled. Disabling this allows the syncs to work, although we still see the exception in the logs (see thread for full stack trace)
Copy code
CONTAINER_ORCHESTRATOR_ENABLED=true
Are there additional environment settings we are missing that need to be added?
Copy code
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
	at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:116) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.29.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
	at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:234) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
	... 3 more
Caused by: io.airbyte.workers.exception.WorkerException: Failed to create pod orchestrator-repl-job-81006-attempt-0, pre-existing pod exists which didn't advance out of the NOT_STARTED state.
	at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:198) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
	... 3 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.70.128.1/api/v1/namespaces/airbyte-dev/pods>. Message: Pod "orchestrator-repl-job-81006-attempt-0" is invalid: [spec.volumes[1].secret.secretName: Required value, spec.containers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.containers[0].volumeMounts[1].mountPath: Required value, spec.initContainers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.initContainers[0].volumeMounts[1].mountPath: Required value]. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.volumes[1].secret.secretName, message=Required value, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec.containers[0].volumeMounts[1].name, message=Not found: "airbyte-secret", reason=FieldValueNotFound, additionalProperties={}), StatusCause(field=spec.containers[0].volumeMounts[1].mountPath, message=Required value, reason=FieldValueRequired, additionalProperties={}), StatusCause(field=spec.initContainers[0].volumeMounts[1].name, message=Not found: "airbyte-secret", reason=FieldValueNotFound, additionalProperties={}), StatusCause(field=spec.initContainers[0].volumeMounts[1].mountPath, message=Required value, reason=FieldValueRequired, additionalProperties={})], group=null, kind=Pod, name=orchestrator-repl-job-81006-attempt-0, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "orchestrator-repl-job-81006-attempt-0" is invalid: [spec.volumes[1].secret.secretName: Required value, spec.containers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.containers[0].volumeMounts[1].mountPath: Required value, spec.initContainers[0].volumeMounts[1].name: Not found: "airbyte-secret", spec.initContainers[0].volumeMounts[1].mountPath: Required value], metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:555) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:305) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:48) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:318) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:308) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?]
	at io.airbyte.workers.process.AsyncOrchestratorPodProcess.create(AsyncOrchestratorPodProcess.java:438) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
	at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:190) ~[io.airbyte-airbyte-commons-worker-0.40.29.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.29.jar:?]
	... 3 more
p
Hello ! I know it's been a while but I have the exact same error message when I'm enabling pod orchestrator on my kubernetes airbyte helm deployment. Did you eventually figured out how to correct this ?
s
Just resolved the problem on my side yesterday. This is linked to a new feature introduce in recent airbyte version (Container Orchestrator) ->Workers &amp; Jobs | Airbyte Documentation
If you are on kube and have gcp bucket, you will have to setup new env variable to your worker pods
p
Hey @Sebastien HO thanks for your reply ! I found a solution as well yesterday night. Had to use the following environment variables in my worker config
Copy code
- name: "CONTAINER_ORCHESTRATOR_SECRET_NAME"
      value: "gcs-log-creds"
- name: "CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH"
      value: "/secrets/gcs-log-creds-worker"
šŸ‘ 1
s
Normally, to make your worker pods connection to your GCP bucket, you have set a kube secret with the gcp json auth information and load this secret as a volume to your worker pod, so that it is accessible. You should have env variable GOOGLE_APPLICATION_CREDENTIALS, this contains the local pod (worker) path to access your mounted GCP secret. Knowing that, in kube •
CONTAINER_ORCHESTRATOR_SECRET_NAME
: it should contains the secret to connect to gcp •
CONTAINER_ORCHESTRATOR_SECRET_MOUNT_PATH :
path where to mount the secret gcp on local orchestrator variable -> it should reflect the GCP
šŸ™ 2