Hi Airbyte team, do you guys plan to provide us a way to set normal webhook notifications for syncs (with a normal JSON payload) instead of just using slack or customerio for example? We have a big application running airbyte and we need to keep track of the sync statuses but we are struggling with the webhook notification settings that are available now.

I can't comment on what their plans are but we use the Airbyte metrics (shipped to Datadog) to keep track of syncs in addition to the webhook notification.

I will take a look at this @Joey Benamy, thank you for your feedback!

You're welcome. Check out https://docs.airbyte.com/operator-guides/collecting-metrics

Hi @Joey Benamy, @Matheus Barbosa, we’ll be working on that very soon. To give you an overview of what I have in mind. In the current webhook notifications we add a

data

node with a json dictionary containing detailed information about the event. It would be very similar to what we send to customerio. e.g for a job failure:

{
  ...
  "data": {
    "workspace": {
      "id": "69de1f06-4fc0-44b0-9592-4cb72ef1c211",
      "name": "workspace-name",
      "url": "<http://link/to/workspace>"
    },
    "connection": {
      "id": "a9b541c1-22a2-44f3-b6b0-98663a82e657",
      "name": "connection-name",
      "url": "<http://link/to/connection>"
    },
    "source": {
      "id": "196757e2-511d-4430-a0a8-1343869d267d",
      "name": "source-name",
      "url": "<http://link/to/source>"
    },
    "destination": {
      "id": "47cd577f-088e-49db-8125-204aae6af7b0",
      "name": "destination-name",
      "url": "<http://link/to/destination>"
    },
    "jobId": 100,
    "startedAt": "1970-01-12T13:46:40Z",
    "finishedAt": "1970-01-13T09:13:20Z",
    "bytesEmitted": 1000,
    "bytesEmittedFormatted": "1000 B",
    "bytesCommitted": 9000,
    "bytesCommittedFormatted": "8 kB",
    "recordsEmitted": 50,
    "recordsCommitted": 48,
    "errorMessage": "Connection to the source failed",
    "durationInSeconds": 70000,
    "durationFormatted": "19 hours 26 min",
    "success": false
  }
  ...
}

Let me know if this is what you would expect here.

That is good in my opinion. It would also be nice if we could send the webhook notification to multiple places or if airbyte can log the webhook notification to stdout as well then i can ingest it into Datadog (which you can't send wbehooks to)

Yeah this way would be perfect @Malik Diarra (Airbyte)! That’s exactly what we are looking for! Do you guys have some idea of when will it be available? We are really struggling in handle these webhooks effectively as it’s not structured yet