https://linen.dev logo
a

Aldo M

10/08/2021, 8:07 AM
Hello everyone ๐Ÿ™‚ I am curious to know when you plan to integrate Prefect ?
๐Ÿ”ฅ 5
m

Michel

10/08/2021, 3:48 PM
Hey @Aldo M, great question. In your mind what would such an integration look like?
m

Matt Arderne

10/10/2021, 4:08 PM
Something like this would be a good start. I'm considering the simplest way to orchestrate a few ELT jobs, I don't like the complexity of airflow and quite liked the experience of using Prefect in the past https://docs.airbyte.io/operator-guides/using-the-airflow-airbyte-operator
a

Aldo M

10/11/2021, 7:46 AM
Hey @Michel I join Matt and I am quite jealous you developed an integration with Airflow and not yet for prefect, as Prefect experience is the best for now on my opinion.
m

Michel

10/11/2021, 9:57 PM
Terrific we just had a convo with their team ๐Ÿ™‚. We can create a prefect task by following the model of the one we have for Airflow I think the v0 could be out fairly quickly, especially if an expert from the community wants to take a stab at it ๐Ÿ™‚
@Aldo M @Matt Arderne would you be interested? We can jump on a call and discuss it some day. I can also bring @[DEPRECATED] Marcos Marx who built the aiflow operator
a

Aldo M

10/12/2021, 1:22 PM
I think the best way to implement it on prefect will be through an API, as Prefect doesnโ€™t have operators like Airflow.. I am in the process of learning settings of Airbyte, and I would like to know if youโ€™ve already have a Trigger API for launching Sync with python script?
u

[DEPRECATED] Marcos Marx

10/12/2021, 2:40 PM
I am in the process of learning settings of Airbyte, and I would like to know if youโ€™ve already have a Trigger API for launching Sync with python script?ย (edited)
Airbyte has an API to trigger jobs but not a lib/wrapper done in Python. I saw some people using OpenAPI Generator lib to help using Airbyte API with Python. I'll take a look in Prefect ๐Ÿ™‚
Also I created the issue: https://github.com/airbytehq/airbyte/issues/6979 to keep track of this featur ๐Ÿ˜„
๐Ÿ‘ 3
a

Aldo M

10/13/2021, 7:38 AM
@[DEPRECATED] Marcos Marx yes I would have used a OpenAPI or requests to trigger it. But if you could implement a lib for using the Airbyte API in Python, that would be great for Prefect users. I will follow the issue. ๐Ÿ˜
Do you have any idea when you will start working on this or when we could except a first version ๐Ÿ™‚ No pression ๐Ÿ˜‰
m

Matt Arderne

10/13/2021, 2:45 PM
Will follow closely, I'm not using Airbyte just yet but if I do, then it will be with Prefect, so will get involved ๐Ÿ‘
6 Views