Is anyone aware of issues in Airbyte >=0.40.26 ...
# replication-troubleshootingg
Gabriel Levine
01/19/2023, 8:31 PMIs anyone aware of issues in Airbyte >=0.40.26 on Kubernetes (GKE) preventing the airbyte-worker from launching new pods?
Gabriel Levine
01/19/2023, 8:33 PMSeems to be a temporal issue:
Copy code
Error starting Micronaut server: Error instantiating bean of type [io.airbyte.workers.temporal.sync.ReplicationActivityImpl]Gabriel Levine
01/19/2023, 8:37 PMI’ve gone through some common Micronaut issues. My deployment doesn’t use Minio or S3. I use GCS for logging. I’ve also verified that WORKERS_MICRONAUT_ENVIRONMENTS variable is set to
control-plane Copy code
# S3/Minio Log Configuration
S3_LOG_BUCKET=
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=
S3_PATH_STYLE_ACCESS=
# GCS Log Configuration
GCS_LOG_BUCKET=leap-airbyte-prod-logging
# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=
STATE_STORAGE_MINIO_ENDPOINT=
# Worker #
WORKERS_MICRONAUT_ENVIRONMENTS=control-plane
# Launch a separate pod to orchestrate sync steps
CONTAINER_ORCHESTRATOR_ENABLED=true
CONTAINER_ORCHESTRATOR_IMAGE=
# Cloud Storage Configuration
WORKER_LOGS_STORAGE_TYPE=GCS
WORKER_STATE_STORAGE_TYPE=GCSn
Nataly Merezhuk (Airbyte)
01/20/2023, 1:27 PMHi Gabriel! I've seen some users encounter a similar issue before. I'm sure you've seen this, but attaching this here for documentation purposes:
https://discuss.airbyte.io/t/worker-process-of-airbyte-v0-40-9-fails-to-start-on-custom-s3-config/2849
Are you able to get the server logs? Does this happen during deployment?
g
Gabriel Levine
01/30/2023, 9:05 PMHey @Nataly Merezhuk (Airbyte), I downgraded back to 0.40.25 and now it’s working again. I can grab the server logs or the worker logs?
u
user
01/31/2023, 6:24 PMGlad to hear there's a workaround by downgrading! Both logs if you can!
g
Gabriel Levine
02/02/2023, 6:40 PMI’m going to try upgrading our dev environment to the latest release. I’ll send logs if it’s still not working.
l
Lydia Lim Li Han
05/28/2023, 5:09 PM@Gabriel Levine did you manage to upgrade successfully and also use GCS for logs?
g
Gabriel Levine
05/28/2023, 5:27 PM@Lydia Lim Li Han Yes. I switched over to using Helm with Flux to manage my GKE deployment and upgraded and it resolved this issue. In general, I’ve found that “kicking” the cluster (deleting all running pods) will often resolve this issue
6 Views