Hello, My team is using Airbyte to manage thousand...
# replication-ask-ai
l
Hello, My team is using Airbyte to manage thousand of jobs every day. After those jobs are done, other processes are launched. To know if the jobs are finished we used the Airbyte API like (the Airflow sync operator is doing). However, we found it counter productive to make many call to monitor thousand of jobs every day. Is the Airbyte team planned to implement some sort of callback ? I know there is a webhook that can be used (for slack for example). However, we are not sure if it is suitable to use it to trigger following processes at scale.
k
A message from kapa.ai
Based on the information provided in the extracts, Airbyte does support the use of webhooks for notifications, as mentioned in a [previous support conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1676434362601979). This can be used to send a message or alert when a sync is finished. However, it's not explicitly mentioned whether this can be used to trigger subsequent processes at scale. For monitoring jobs, the [Airbyte API](https://airbyte.com/tutorials/create-and-monitor-a-data-pipeline-using-the-airbyte-api) can be used to check the status of a job every 10 seconds until the job is no longer running. This is done by making a GET request to the
/v1/jobs/{jobId}
endpoint. While this method works, it might not be the most efficient for monitoring thousands of jobs as it involves making frequent API calls.
Copy code
csharp //Checks every 10 seconds the status of the job while(jobStatus == "running") {  //Gets status of the job  client = new RestClient("<https://api.airbyte.com/v1/jobs/>" + jobId);//We add the jobId  request = new RestRequest();  request.AddHeader("accept", "application/json");  request.AddHeader("content-type", "application/json");  request.AddHeader("Authorization", "Bearer {YourApiKey}");  dynamic statusResponse = client.ExecuteGet(request);   //Deserialize response to get the status of the job  apiResponse = JsonConvert.DeserializeObject(statusResponse.Content);  jobStatus = apiResponse.status;   //Waits 10 seconds..  Thread.Sleep(10000);   //Prints info in console depending on the status  if (jobStatus == "running")  {  Console.WriteLine("Job running...");  }  else  {  Console.WriteLine("Job Finished...");  Console.WriteLine(" ");  Console.WriteLine("Sync details:");  Console.WriteLine(" JobId:" + apiResponse.jobId);  Console.WriteLine(" Status:" + jobStatus);  Console.WriteLine(" Rows/Bytes Synced:" + apiResponse.rowsSynced + "/" + apiResponse.bytesSynced);  Console.WriteLine(" Sync Start Time:" + apiResponse.startTime);  Console.WriteLine(" Sync Last Updated:" + apiResponse.lastUpdatedAt);  Console.WriteLine(" Sync Duration:" + apiResponse.duration);  } }
As for the implementation of a callback feature, the extracts do not provide any information on whether this is planned by the Airbyte team. For more specific information or requests, it would be best to reach out to the Airbyte team directly through their official communication channels.
l
@kapa.ai How can I set the webhook url on K8S configurations ?
k
A message from kapa.ai
New message text here
l
@kapa.ai Firstly, we don't want to use a slack webhook. Secondly, we would like to set this url through the Airbyte cluster not from the UI. Setting it from the UI isn't very reliable for production environment...
k
A message from kapa.ai
I understand your concern. However, based on the provided extracts, there's no direct information on how to set the webhook URL through the Kubernetes configuration or any other method outside of the UI. Airbyte's documentation or support channels might have more information on this. I recommend checking the [Airbyte documentation](https://docs.airbyte.com/) or reaching out to the Airbyte community on their [Slack channel](https://airbytehq.slack.com) or [discussion forum](https://discuss.airbyte.io) for more specific guidance.
🫠 1