Hi Everyone have synced airbyte with airflow locally however Airbyte #replication-troubleshooting

Hi Everyone, have synced airbyte with airflow loca...

Arjunsingh Yadav

01/03/2023, 8:52 AM

Hi Everyone, have synced airbyte with airflow locally however while running a dag in airflow, it fails with an exception after making a POST api call to airbyte

Copy code

[2023-01-03, 07:08:39 UTC] {base.py:73} INFO - Using connection ID 'airbyte_conn' for task execution.
[2023-01-03, 07:08:39 UTC] {http.py:150} INFO - Sending 'POST' to url: <http://localhost:8001/api/v1/connections/sync>
[2023-01-03, 07:08:39 UTC] {local_task_job.py:159} INFO - Task exited with return code Negsignal.SIGSEGV

The airbyte connection has S3(cloud) -> Postgres(local) connection and works fine when synced manually. I’m not being able to sync it via airflow. Moreover, the API call to 8001(webapp) server of airbyte requires a basic auth but I didnt find any config to put those creds in airflow. The DAG is as follows -

Copy code

from airflow import DAG
from airflow.utils.dates import days_ago
from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator

with DAG(dag_id='airbyte_job',
         default_args={'owner': 'admin'},
         schedule_interval='@daily',
         start_date=days_ago(1)
         ) as dag:

    airbyte_trigger = AirbyteTriggerSyncOperator(
        task_id='airbyte_airflow',
        airbyte_conn_id='airbyte_conn',
        connection_id='****',
        asynchronous=False,
        timeout=3600,
        wait_seconds=3)

Another possible way to sync it is making an API call from airflow -> airbyte without using airflow’s airbyte provider(https://pypi.org/project/apache-airflow-providers-airbyte/) Can somebody give a curl for running the sync on airbyte?

Arjunsingh Yadav

01/03/2023, 10:41 AM

Solved Hadnt dug deep into the source code earlier setting the username, password as ‘’ works Mentioned in Network Security

Copy code

# Proxy Configuration
    # Set to empty values, e.g. "" to disable basic auth
    BASIC_AUTH_USERNAME=your_new_username_here
    BASIC_AUTH_PASSWORD=your_new_password_here

The curl for syncing a connection -

Copy code

curl --location --request POST '<http://localhost:8001/api/v1/connections/sync>' \
--header 'Authorization: Basic Og==' \
--header 'Content-Type: application/json' \
--data-raw '{
    "connectionId": "Airbyte-rocks"
}'

Nataly Merezhuk (Airbyte)

01/09/2023, 3:21 PM

Hi Arjunsingh! Thanks for your patience - I was out last week for the holidays and am catching up with messages now. So glad to hear this has been resolved and thank you so much for the details!

8 Views

Open in Slack

Previous Next