https://linen.dev logo
#connector-development
Title
# connector-development
w

Will Watkinson

03/17/2022, 2:37 PM
My http connection requires custom error handling. I have set up the
should_retry
method accordingly. Based on the logs
should_retry
is evaluating to
True
as expected given line 200 of the logs contains the
DefaultBackoffException
class, but the default retry behavior of retrying 5 times with exponential backoff does not seem to be kicking in. Has the default behavior changed, or is there something additional I need to do to enable it?
a

Augustin Lafanechere (Airbyte)

03/18/2022, 2:13 PM
Hey @Will Watkinson do you mind sharing your stream class code?
a

Augustin Lafanechere (Airbyte)

03/18/2022, 5:03 PM
Could you please try to set
max_retries = 5
in the class attribute to check if anything changes? I dug a bit in the CDK code and I don't think the usual logic changed...
Could you also please share your full sync logs?
w

Will Watkinson

03/18/2022, 5:56 PM
yeah I dug through the code too and it seemed like max_retries should be defaulting to 5 as specified in the
HttpStream
class
ran into the same error with `max_retries` explicitly set to 5
It looks like the
should_give_up
method in
rate_limiting.py
is overwriting my desired behavior. https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/streams/http/rate_limiting.py#L35
a

Augustin Lafanechere (Airbyte)

03/22/2022, 6:12 PM
Nice catch, thanks for digging in @Will Watkinson. I guess you solved this problem as we started another CDK conversation.
w

Will Watkinson

03/22/2022, 6:15 PM
I resolved this temporarily by overwriting the 401 status code with a status code that would be retried. Also opened and issue and put up a PR to resolve it in the CDK. I couldn't sign the CLA, though
2 Views