Hello, is there a way to disable the 3 retries when a connection fails? I have a connection that often fails because of the API rate limits and retrying is only making this worse.

Not today. But this is a great idea. Can you open an issue for this? We can inject this pretty easily

Sure! https://github.com/airbytehq/airbyte/issues/4818

Jaime, this is easy to do as a blanket setting but more difficult to do for individual connections. Would that satisfy your use case?

Mmmm no worries for my specific case. My colleague has developed a fix for the rate limit of that connection: https://github.com/airbytehq/airbyte/pull/4820

Develop it as you think it’s easier or fixes the general use case

Thanks for the contribution!

Thanks for building Airbyte, it’s great!

are you finally developing the capacity to set up retries?

what do you mean?

the solution to this issue: https://github.com/airbytehq/airbyte/issues/4818

I see. No timeline yet - we are taking it into consideration. Is it blocking you?

Mmmmm not really, but I see a weird behavior with retries and Airflow connector

It would be easier for me managing the retries from Airflow with new jobs on Airbyte

@Jaime Farres maybe something more easy (as a workaround) is choose the # of attempts with a env variable. So you could choose 1 and control the retry using Airflow DAG, WDYT?

Yes, that absolutely will work

Do you think this is something you will work on soon?

Maybe I can work on this. It would be changing: • 151 at https://github.com/airbytehq/airbyte/blob/master/airbyte-workers/src/main/java/io/airbyte/workers/temporal/SyncWorkflow.java • Adding new env variable with default 3 to https://github.com/airbytehq/airbyte/blob/master/.env Am I right? Something else?

@Jaime Farres I think this is the way!

I think I cannt work on this soon, but looks a easy addition if you could do that will be awesome!

one question, MAX RETRIES = 3 in line 151 at https://github.com/airbytehq/airbyte/blob/master/airbyte-workers/src/main/java/io/airbyte/workers/temporal/SyncWorkflow.java it refers to the number of attempts or to the number of retries inside an attempt?

I think is the latter, and we need to change the MAX_ATTEMPTS, which I’m not able to find

do you feel comfortable to edit and test with

dev

build?

yes, mi colleague @Vladimir Remar is trying

we have reached the conclusion that MAX_RETRIES applies both to: • number of retries within an attempt • number of attempts

No, this conclusion was wrong, we have made a dev build with MAX_RETRIES = 2 and: • number of retries within an attempt : 2 • number of attempts : 3

if you could help us understand where the amount of attempts is set up that would be great

this is exposed in the JobRetrier class

Thanks!

Hello guys I made this PR if you can check it I will appreciate.