<@U04S8PLJWMT> I've setup Airbyte in my eks cluste...
# ask-ai
j
@kapa.ai I've setup Airbyte in my eks cluster, slack notifications work when I click Test in the UI but they don't when a stream actually succeeds or fails even though these events are enabled in the UI.
k
Based on the docs and issues you shared, there are a few key points relevant to what you’re seeing: 1. Webhook/Slack path is working The fact that the Test button sends a message to Slack confirms: - Your Slack app and incoming webhook URL are correct. - Airbyte can reach Slack from your EKS cluster. [[Configure Slack](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#configure-slack-notifications)] 2. Successful/failed sync webhooks should fire with payloads For both Cloud and “Next” platform docs, a real sync (success or failure) is supposed to send a JSON payload like the examples shown for Failed sync and Successful sync to the configured webhook. [[Set up webhook](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-webhook-notifications)] 3. Some notification types don’t send payloads / only work for Slack-rendered messages Several notification types explicitly “don’t contain payload and only work for Slack notifications”, such as: - Connection updates - Connection updates requiring action - Warning / Sync disabled due to repeated failures - Upgrade required / Sync disabled – upgrade required [[Set up webhook](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-webhook-notifications)] However, for sync success/failure, the docs are clear that webhook notifications do send payloads and are intended to work with generic webhooks (including Slack). [[Set up webhook](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-webhook-notifications)] 4. There was a known bug affecting (at least) successful sync webhooks A GitHub issue notes: “There was a bug with the successful sync webhook code,” which was then fixed. [[Webhook issue](https://github.com/airbytehq/airbyte/issues/62117)] The issue does not specify versions, so it’s unclear whether your EKS deployment contains that fix. Given the sources, there are two likely explanations consistent with the docs: - Your deployment might be on an Airbyte version that still has the successful-sync webhook bug, so only test messages work. - You may be primarily testing notification types that docs state only work as Slack-rendered notifications (no generic webhook payload), though for plain success/failure of a sync they should be emitted. Because the knowledge sources do not list a concrete version matrix or specific EKS/self‑hosted behavior, I cannot definitively say which version you need to upgrade to, or whether your exact build contains the fix. What you can do next (based only on the docs/issues): 1. Confirm which notification type is enabled Ensure you have the Failed sync and Successful sync (or equivalent naming in your UI) toggles enabled under Settings → Notifications → Webhook for the workspace/connection you’re testing. [[Enable Slack](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#enable-the-slack-notification-in-airbyte)] 2. Check your Airbyte version against the bug Since there was a bug in successful-sync webhooks that’s now “fixed,” verify: - The exact Airbyte version running in your EKS cluster. - Whether your image/tag includes the fix mentioned in the webhook GitHub issue. [[Webhook issue](https://github.com/airbytehq/airbyte/issues/62117)] 3. If you’re on an older version, upgrade The docs imply that the expected behavior is: - Test button → Slack message - Real sync success/failure → Slack message with the documented JSON payload If you’re not seeing that, and since we know there was at least one bug in this area, upgrading to a more recent version is the most aligned step with the information available. 4. If after upgrading it still doesn’t trigger The provided sources don’t contain further troubleshooting steps specific to self‑hosted/EKS beyond configuration and the known bug. At that point, the next action—based on the docs and GitHub patterns—would be to open an issue or discussion with: - Your Airbyte version - Confirmation that Test works - Logs from the worker/server around the time of the sync Because the documentation and issues you shared don’t go deeper into this specific failure mode on EKS/self‑hosted, I can’t provide more detailed debugging steps without going beyond the available sources.
j
@kapa.ai I'm using version 1.7.8, and all the notifications are enabled. Yesterday I got some notifications but they were sent a specific time (8:02pm and 9:02pm) as if they were triggered by a cron and not by when the job finished and this morning I did not get any notification after completing another stream.
k
A message from kapa.ai
On the versions documented (1.6–1.8 and Next), notifications are described as being tied to sync events, not to a cron-like schedule. For real syncs, Airbyte is expected to send a webhook payload right after the job completes, with fields like
success
,
startedAt
,
finishedAt
,
durationInSeconds
, etc. [[Webhook payload](https://docs.airbyte.com/platform/1.7/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#configure-webhook-notification-settings)] The knowledge sources you provided do not mention: - Any intentional “batching” or fixed-time emission of notifications (e.g., always at
:02
past the hour). - Any change in 1.7.x that would cause notifications to be driven by a cron rather than by job completion time. - Any known 1.7.8-specific bug where some syncs produce notifications and others don’t. The only relevant bug we see is a generic one: “There was a bug with the successful sync webhook code,” which was then fixed, but without version details. [[Webhook issue](https://github.com/airbytehq/airbyte/issues/62117)] It’s unclear from the available sources whether 1.7.8 contains that fix or is affected by some related regression. Because of that, I can’t definitively explain the “always at 802/902pm” behavior or why this morning’s run produced no notification. The documentation simply doesn’t describe any mechanism that would schedule notifications independently of job completion, nor does it give a troubleshooting flow for intermittent notification delivery. Given the limits of the docs/issues you shared, the most I can say confidently is: - Expected behavior (per docs): each successful/failed sync should immediately emit a webhook with the JSON payload shown for “Successful Sync” / “Failed Sync.” [[Webhook payload](https://docs.airbyte.com/platform/1.7/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#configure-webhook-notification-settings)] - Your observed behavior (cron-like times, then no notification) is not described anywhere in these knowledge sources. - There is at least one historical bug around successful sync webhooks, but the sources don’t state whether 1.7.8 is impacted. [[Webhook issue](https://github.com/airbytehq/airbyte/issues/62117)] To go further than this (e.g., confirm if 1.7.8 has a known bug or suggest a concrete patch or config change) would require information that is not present in the documents you provided, so I’d be speculating beyond the available sources.
j
@kapa.ai what pods should I check the logs in case the webhooks fail?
k
A message from kapa.ai
On Kubernetes, the only guidance in the provided sources is generic log access; there is no explicit list of which pods handle webhook notifications or Slack in particular. The docs simply show how to exec into and copy logs from pods in general, by targeting the pod you are interested in and using the
main
container: [[Browsing logs](https://docs.airbyte.com/platform/1.7/operator-guides/browsing-output-logs#browsing-on-kubernetes)]
Copy code
bash # Exec into a pod kubectl exec -it <pod-name> -n <namespace> -c main bash  # Example from docs: kubectl exec -it destination-bigquery-worker-3607-0-chlle -n jobs -c main bash
From the available documentation, I can only say: - You should check logs in the pods that are responsible for running jobs and handling platform functionality, but - The knowledge sources do not specify which exact pod names/components in a 1.7.x deployment are responsible for emitting webhook notifications. So, within the constraints of these docs, I can’t name specific pods beyond “the pod you are interested in” and the example worker pod shown above. [[Browsing logs](https://docs.airbyte.com/platform/1.7/operator-guides/browsing-output-logs#browsing-on-kubernetes)]