Hi everyone, I wanna ask about Dagster+Airbyte com...
# replication-troubleshooting
y
Hi everyone, I wanna ask about Dagster+Airbyte combo. We have configured those two and it has been operating until now. But we often receive “Exceeded max number of retries” in Dagster-Airbyte job (I am sure its called asset materialization). So we can’t fully depends on Dagster trigger/sensor mechanism based on Airbyte asset materialization in Dagster (context, we want to run some Dagster operation/job after certain Airbyte asset has been updated). So, is there any workaround for this thing? Configuration & Specs Dagster 1.0.9 Airbyte 0.40.28 request_max_retriex : 15 (Dagster-Airbyte Asset) request_retry_delay : 2 (Dagster-Airbyte Asset) We use default value for all Airbyte limit env
u
Hello Yogic Wahyu, it's been a while without an update from us. Are you still having problems or did you find a solution?
y
Hi @Saj Dider (Airbyte), we still get failed metadata retrieval from
Airbyte
pipeline connection/scheduler in
Dagster
. We have increased backoff time and retry limit but still failed. It’s unfortunately that we cant create
Dagster
sensor depends on
Airbyte
materialization.