Enrico Tuvera Jr.
03/18/2022, 6:37 AMAugustin Lafanechere (Airbyte)
03/18/2022, 2:12 PMHttpStream.should_retry
function will return True and will use the value returned by YourStream.backoff_time
to sleep before next request.
In other words: you need to implement a backoff_time
function in your stream class to return the desired sleep time. If your API 429 responses provides X-Retry-After
it can help you dynamically define the sleep time. If your API does not use 429 statuses to inform of the rate limiting you need to also implement a custom should_retry
function. Feel free to check the implementation of the base HttpStream class to understand how it works by default.