Luke Parker
10/23/2024, 9:50 AMuser
10/23/2024, 9:50 AMpython from airflow import DAG from airflow.utils.dates import days_ago from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator with DAG(dag_id='trigger_airbyte_job_example', default_args={'owner': 'airflow'}, schedule_interval='@daily', start_date=days_ago(1) ) as dag: money_to_json = AirbyteTriggerSyncOperator( task_id='airbyte_money_json_example', airbyte_conn_id='airbyte_conn_example', connection_id='1e3b5a72-7bfd-4808-a13c-204505490110', asynchronous=False, timeout=3600, wait_seconds=3 )
[Using the Airbyte Operator to orchestrate Airbyte OSS](https://docs.airbyte.com/operator-guides/using-the-airflow-airbyte-operator#creating-a-simple-airflow-dag-to-run-an-airbyte-sync-job) 2. Using Dagster: Dagster is another orchestration tool that supports Airbyte integration. You can create a Dagster job to trigger Airbyte syncs: python from dagster import job from dagster_airbyte import airbyte_resource, airbyte_sync_op my_airbyte_resource = airbyte_resource.configured( { "host": {"env": "AIRBYTE_HOST"}, "port": {"env": "AIRBYTE_PORT"}, } ) sync_foobar = airbyte_sync_op.configured({"connection_id": "your-connection-uuid"}, name="sync_foobar") @job(resource_defs={"airbyte": my_airbyte_resource}) def my_simple_airbyte_job(): sync_foobar()
[Using the Dagster Integration](https://docs.airbyte.com/operator-guides/using-dagster-integration#2-create-the-dagster-op-to-trigger-your-airbyte-job) 3. Using Prefect: Prefect is another workflow management system that can trigger Airbyte jobs: python from prefect import Flow from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask airbyte_conn = AirbyteConnectionTask( airbyte_server_host="localhost", airbyte_server_port=8000, airbyte_api_version="v1", connection_id="04e128af-1092-4a83-bf33-1b8c85395d74" ) with Flow("first-airbyte-task") as flow: flow.add_task(airbyte_conn)
[Using the Prefect Airbyte Task](https://docs.airbyte.com/operator-guides/using-prefect-task#2-create-a-flow-in-prefect-to-trigger-your-airbyte-job) These methods allow you to trigger Airbyte jobs programmatically using Python within different orchestration frameworks. Choose the one that best fits your existing infrastructure and requirements.