How to Run Synchronization of a Connection in AirByte Using Python in an AirFlow Task without AirbyteOperator

To run synchronization of a connection in Airbyte using Python in an Airflow task without AirbyteOperator, you can use the Airbyte's API directly. Here is a general idea of how you can do it: 1. First, you need to create an Airflow task that makes a POST request to the Airbyte's

/v1/connections/sync

endpoint. This endpoint triggers a manual sync of the connection. The payload of the request should be a JSON object containing the

connectionId

of the connection you want to sync.

python import requests  def trigger_airbyte_sync(connection_id):     url = "<http://localhost:8000/api/v1/connections/sync>"     headers = {"Content-Type": "application/json"}     data = {"connectionId": connection_id}     response = <http://requests.post|requests.post>(url, headers=headers, json=data)     if response.status_code != 200:         raise Exception(f"Failed to trigger sync: {response.text}")     return response.json()

2. Then, you can create an Airflow task that waits for the sync to complete. You can do this by periodically making a GET request to the

/v1/jobs/get

endpoint of the Airbyte's API, and checking the status of the job.

python import time  def wait_for_airbyte_sync(job_id):     url = f"<http://localhost:8000/api/v1/jobs/get/{job_id}>"     while True:         response = requests.get(url)         if response.status_code != 200:             raise Exception(f"Failed to get job status: {response.text}")         status = response.json()["job"]["status"]         if status == "succeeded":             break         elif status == "failed":             raise Exception("Job failed")         time.sleep(10)  # wait for 10 seconds before checking the status again

Please note that this is a simplified example and you might need to adjust it according to your needs and environment. Also, replace

"<http://localhost:8000>"

with the actual URL of your Airbyte server. For more details, you can refer to the [Airbyte API documentation](https://docs.airbyte.io/understanding-airbyte/airbyte-api). Knowledge Source(s): - [Github Issue (open): Self-hosted Airbyte in Docker stuck and will not trigger sync > Original post Part 3](https://github.com/airbytehq/airbyte/issues/44833) - [airbyte-server > One Example of API workflow Part 1](https://github.com/airbytehq/airbyte/discussions/33811)

The Get call /api/v1/jobs/get/{job_id} is 404