https://linen.dev logo
Join Slack
Powered by
Greetings, all. I’ve been attempting to run AirByt...
# replication-troubleshooting
s
Greetings, all. I’ve been attempting to run AirByte over the weekend but ran into some sync issues. Above is the log messages I’ve been getting from the AirByte Worker. There have been restarts on cron, the server, the worker, and the webapp. Does anyone have any pointers as to why this might be happening?
Untitled.txt
On second glance, perhaps this is an issue with my Kubernetes deployment. I will also open a ticket w/ Linode.
m
What version of Airbyte are you deploying? Are you using kustomize or Helm?
s
I’m currently on 0.40.24, deployed via Helm.
It appears at this point that AirByte is not able to start up new pods at all. I see no k8s output to indicate success or failure either way, and I’m not sure how to troubleshoot deeper into k8s.
Update: I killed the worker and temporal pods and now sync operations at least can start up one or two pods. The write pod comes up right away; the read pod remains pending for quite a while. Still waiting to see if the read pod comes up at all.
I’ve killed existing pods to test that the cluster itself can start up pods. All works as expected. It seems AirByte specifically cannot seem to start up pods for whatever reason.
Another update: it appears busybox cannot be initialized. Here is the error from the logs:
Copy code
Timeout while attempting to copy to init container, exiting with code 1...
Does this look familiar to anyone? Could this be a disk space issue?
I see Issue #20120 relates to this.
m
@Sean Zicari strange because the init container should exit during the bootloader process
are you using a different namespace than default? any changes to values in helm?
s
I put AirByte in an 
airbyte
namespace. In l configured the image pull secret, but otherwise no customizations.
Which boot loader process? It looks like it never gets started.
m
whenever you start airbyte, the init container must exit after finish the initial tasks
s
Hm. AirByte has been running for days. I’m seeing these failures when starting up sync tasks.
Here’s a screenshot of the pods inside one of the sync jobs. The consistent part of each failed sync job is the Error at the busybox container.
Do you recommend I install it in the default namespace?
FYI - I temporarily worked around this issue by increasing the cluster node count from 2 to 3. I was thinking of increasing to 3 nodes, anyway, and this seemed like a good time to do so. It actually did (temporarily?) resolve the issues.
👍 1
Revisiting a few days later: the sync process is still working!
u
Nice, do you need any further assistane here Sean?
s
I do not believe so. Thank you again.
2 Views
Open in Slack
PreviousNext
Powered by