https://linen.dev logo
Join Slack
Powered by
Any clue on how to debug this? When trying to set ...
# replication-troubleshooting
r
Any clue on how to debug this? When trying to set a destination or source to a different version in the UI I get 
error, something went wrong
with the following logs in the server pod:
Copy code
POST 500 /api/v1/destination_definitions/update - {"destinationDefinitionId":"22f6c74f-5699-40ff-833c-4a879ea40133","dockerImageTag":"1.2.7"}
e
Hi. 500 is server error- did you turn on docker?
r
I'm using the helm deployment (kubernetes on GKE)
e
were you able to connect before to the api?
r
Yeah, I used to be in an older helm chart and everything was working fine
In the worker pod I see the following logs:
Copy code
Caused by: io.temporal.failure.ApplicationFailure: message='io.airbyte.workers.exception.WorkerException: Error while getting spec from image airbyte/destination-bigquery:1.2.7', type='java.util.concurrent.ExecutionException', nonRetryable=false at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?] at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.23.jar:?] at io.airbyte.workers.temporal.spec.SpecActivityImpl.run(SpecActivityImpl.java:102) ~[io.airbyte-airbyte-workers-0.40.23.jar:?] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?] at java.base/java.lang.reflect.Method.invoke(Method.java:578) ~[?:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?] at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.base/java.lang.Thread.run(Thread.java:1589) ~[?:?] Caused by: io.temporal.failure.ApplicationFailure: message='Error while getting spec from image airbyte/destination-bigquery:1.2.7', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:96) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:37) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.23.jar:?] at java.base/java.lang.Thread.run(Thread.java:1589) ~[?:?] Caused by: io.temporal.failure.ApplicationFailure: message='Failure executing: POST at: <https://10.36.0.1/api/v1/namespaces/default/pods>. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:77) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:61) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:37) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.23.jar:?] at java.base/java.lang.Thread.run(Thread.java:1589) ~[?:?] Caused by: io.temporal.failure.ApplicationFailure: message='Failure executing: POST at: <https://10.36.0.1/api/v1/namespaces/default/pods>. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.', type='io.fabric8.kubernetes.client.KubernetesClientException', nonRetryable=false at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:610) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:555) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:518) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:305) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.utils.CreateOrReplaceHelper.createOrReplace(CreateOrReplaceHelper.java:48) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:318) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:308) ~[kubernetes-client-5.12.2.jar:?] at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:83) ~[kubernetes-client-5.12.2.jar:?] at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:523) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.spec(AirbyteIntegrationLauncher.java:77) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:61) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.general.DefaultGetSpecWorker.run(DefaultGetSpecWorker.java:37) ~[io.airbyte-airbyte-commons-worker-0.40.23.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.40.23.jar:?] at java.base/java.lang.Thread.run(Thread.java:1589) ~[?:?]
e
sounds odd- did you make sure all the settings are the same and correct?
ok- based on the error there is an issue while fetching the image from bq. look to see if all is good there - sounds very odd - this shouldn't happen
r
Hmm, in the logs
Copy code
<https://10.36.0.1/api/v1/namespaces/default/pods>. Message: Unauthorized! Configured service account doesn't have access.
e
yea either you dont have access or it doesnt exists (shouldn't happen)
r
Maybe something in the charts values.yaml changed around setting the SA?
e
Copy code
Service account may have been revoked. Unauthorized
check the SA
hard to say
check the SA to see if all is good there and if yes try to redeploy and see if you used the correct SA
✅ 1
r
Currently it's being set by
Copy code
serviceAccount:
  annotations:
    {
      <http://iam.gke.io/gcp-service-account|iam.gke.io/gcp-service-account>: ${airbyte_sa}
    }
in the values.yaml (${airbyte_sa} is actually injecting it)
e
hhh very odd - check the SA
r
Hmm, after redeployment it is working again, no clue what went wrong
e
some bug
n
@Ramon Vermeulen glad to hear everything's working, and @Eli Sigal thanks so much for being such an active part of the community 🙂
octavia loves 1
j
Hey there, I’m facing the same problem. I do believe it’s related to https://github.com/airbytehq/airbyte/issues/18731. Thats explain why, after redeploying, OP could make it work..
👍 1
4 Views
Open in Slack
PreviousNext
Powered by