Recently we've moved our OSS Airbyte installation ...
# connector-contributions
m
Recently we've moved our OSS Airbyte installation from Airbyte 0.63.15 running on docker compose to 1.0.0 running on kubernetes. We've got a few dozens of connectors running every day and usually they start and finish successfully, but sometimes it happens that a connector stucks in "Running" state forever and it's replication job pod is missing. In logs I found that the destination process exited with non-zero exit code 143. Is this a bug in connector process monitoring? Full log message in the🧵.
message has been deleted
Another time I saw a job that was still in Running state but the repliction job pod was not running and there weren't any messages saved in logs. Is this possible that if the kubernetes node is terminated unexpectedly Airbyte still thinks the pod is running?
I think it might be related to AWS Scalling group actions that may terminate an instance and start a new one instead:
Copy code
At 2024-10-09T17:07:59Z availability zones us-east-2a us-east-2b had 0 2 instances respectively. An instance was launched to aid in balancing the group's zones.

At 2024-10-09T17:07:59Z instances were launched to balance instances in zones us-east-2a us-east-2b with other zones resulting in more than desired number of instances in the group. At 2024-10-09T17:08:09Z an instance was taken out of service in response to a difference between desired and actual capacity, shrinking the capacity from 4 to 3. At 2024-10-09T17:08:09Z instance i-0897ed159ca2b505b was selected for termination.
Does airbyte has any replication job monitoring in place?
n
Yeah, sounds suspiciously like a cluster resource / autoscaler rules problem. But, that’s outside of my area of expertise, so ;( Is it feasible for you to make consistent node pool and resourcing and disable autoscaling?
m
The whole point of us moving to kubernetes was to implement automatic autoscalling. Perhaps I can limit the auto scalling group to a single AWS availability zone to eliminate the balancing issue, but this is not a real solution as nodes can fail and be replaced anyway.
Airbyte should be able to recover jobs from failed nodes automatically. It can recover from failed source or destination processes, but it looks like it can't recover from a failed orchestrator 🤔 or the whole replication job pod.