Hi team, I’m running Airbyte 0.40.25 on GKE and my...
# replication-troubleshooting
g
Hi team, I’m running Airbyte 0.40.25 on GKE and my installation is failing to launch new pods. I thought it might be a temporal issue, so I upgraded temporal to
airbyte/temporal-auto-setup:1.13.0
. But am still having the same issue.
Copy code
2022-12-16 20:31:09 - Additional Failure Information: scheduledEventId=68, startedEventId=69, activityType='RunWithJobOutput', activityId='fef2fc91-f9b5-3935-a79a-e5fb5ce55fc9', identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
m
Gabriel this is a new deployment or it was running for a while and started failing? Are all services running correctly?
g
Hey @Marcos Marx (Airbyte). This was an existing deployment running for awhile that started failing shortly after an upgrade to 0.40.25. I fixed it by increasing the temporal activity max timeout:
ACTIVITY_MAX_TIMEOUT_SECOND=600
in the k8s configuration. It was timing out because I’m on an autoscaling cluster and it wasn’t giving new nodes sufficient time to launch. It might be worth documenting the timeout variable as it’s not in the existing example manifests.
f
@Gabriel Levine hi, where did this change ACTIVITY_MAX_TIMEOUT_SECOND=600?
g
It’s set here but that’s not documented anywhere
u
There is this doc https://docs.airbyte.com/operator-guides/configuring-airbyte but is not updated :(