Backoff before next attempt: 10 seconds 2024-10-25 224437 ERROR i.a.w.l.p.h.FailureHandler(apply):39 - Pipeline Error io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: CHECK pod failed to init within allotted timeout. It seems when deploying to kubernetes, I get tolerations on some of the pods, not on connectors - they start without tolerations. I am on 1.1.0 Airbyte and it was upgraded from 0.48.

Based on the information provided in the knowledge sources, it appears that you are experiencing a known issue with Airbyte on Kubernetes where tolerations are not being properly applied to all pods, particularly the connector pods. This issue has been reported and discussed in multiple GitHub issues. Here's a summary of the situation: 1. The problem you're describing is consistent with the issue reported in [GitHub Issue #28389](https://github.com/airbytehq/airbyte/issues/28389) and [GitHub Issue #45903](https://github.com/airbytehq/airbyte/issues/45903). These issues describe that job tolerations are being ignored for certain pods, including orchestrator and connector pods. 2. This issue has persisted across multiple versions of Airbyte, including the version you're using (1.1.0). As noted in one of the comments, "1.1.0 - still there." [Source](https://github.com/airbytehq/airbyte/issues/28389) 3. The problem affects the scheduling of pods, as they cannot be scheduled on nodes with specific taints due to the missing tolerations. This can lead to pods failing to start or being scheduled incorrectly. 4. There have been attempts to fix this issue, but it seems the fix has not been fully implemented or released yet. A potential fix was mentioned in a commit, but it appears it hasn't resolved the issue for all users. [Source](https://github.com/airbytehq/airbyte/issues/28389) 5. As a workaround, some users have suggested using a Kyverno policy to automatically add the required tolerations to pods based on certain conditions. However, this is not an official solution from Airbyte. [Source](https://github.com/airbytehq/airbyte/issues/28389) 6. The Airbyte team is aware of this issue and it has been raised to the deployment team, but as of the latest information available, a complete fix has not been released. [Source](https://github.com/airbytehq/airbyte/issues/28389) Given that you've upgraded from 0.48 to 1.1.0 and are still experiencing this issue, it's consistent with the reports from other users. Unfortunately, there doesn't seem to be an official fix available yet. You may want to consider using the Kyverno policy workaround if it's suitable for your environment, or wait for an official fix in a future release. For the most up-to-date information on this issue and potential fixes, I recommend following the GitHub issues mentioned above and checking for any new releases or announcements from the Airbyte team.