adding the recent shoutouts https://twitter.com/dagster/status/1579902784438435840

This channel is a place where you can get the latest updates on using Dagster with Airbyte, ask questions, and discuss with other members of the community. Looking to get started using Dagster with Airbyte? • ✨ Check out our getting started with Airbyte guide • 📚 Take a look at our API docs for `dagster-airbyte` • S Join the Dagster Slack for more discussion about the general framework!

I tested the dagster new feature to manage Airbyte's artifact with Python code.

reminder that our community call starts in 30 mins! https://airbytehq.slack.com/archives/C01AHCD885S/p1673987146969639

Q&A welcome!

When I'm using the

load_assets_from_airbyte_instance

method to create the assets from my Airbyte connections, and I have the connections set to run on a schedule from the Airbyte side, does anyone know if Dagster will automatically detect when the assets get materialized via an Airbyte initiated sync, or does it have to be triggered from the Dagster side?

An issue that I'm experiencing with my self-hosted Airbyte instance that has multiple connections is that, with a multiprocess executor Dagster will try to trigger multiple syncs simultaneously, and several of them will fail with a 409 error, which appears to be related to https://github.com/airbytehq/airbyte/issues/21208. One thought I had was to try introducing some skew to the API calls from the Dagster integration, but I'm not seeing any built-in way to do that. Is there some option that folks have used in the multiprocess executor to do something similar?

After upgrading to 0.41.0 my Dagster pipeline doesn't seem to be able to enumerate the list of Airbyte connections now. The pipeline shows as being empty (relevant code is here: https://github.com/mitodl/ol-data-platform/blob/main/src/ol_orchestrate/definitions/elt.py#L43-L54)

Is there an update to the Airbyte API that might have resulted in this happening which necessitates a code change in the dagster-airbyte plugin?

Scratch that. It appears that it was actually the latest Dagster release that is resulting in the empty pipeline.

archived the channel