How can I execute the exact same api call once again in an httpStream? Basically my api endpoint creates a csv-file that is accessible via a url the api returns. at the first call the staus is "in progress" so the csv is still being created. When I call the api a second time (e.g. via postman) the status is changed to "done" an the endpoint returns an url

To execute the same API call again in an

HTTPStream

, you can use the retry mechanism provided by the Airbyte CDK. By default, the CDK will conduct exponential backoff on the HTTP code 429 and any 5XX exceptions, and fail after 5 tries. This retry mechanism is governed by the

should_retry

and the

backoff_time

methods. You can override these methods to customize the retry behavior according to your needs. In your case, you can customize the

should_retry

method to retry the request when the status is "in progress". Here's a pseudo-code example:

python def should_retry(self, response: requests.Response) -> bool:     # Parse the response     parsed_response = self.parse_response(response)     # Check if the status is "in progress"     if parsed_response['status'] == 'in progress':         return True     return False

Please note that this is a pseudo-code example and you might need to adjust it according to your actual API response structure. For more information, you can refer to the [Airbyte documentation on HTTP Streams](https://docs.airbyte.com/connector-development/cdk-python/http-streams).

@kapa.ai is it possible for my custom source to return that csv-file? Pandas can read that file but how do I return it in my HttpStream?

Yes, it is possible for your custom source to return a CSV file. You can use the

HTTPStream

object provided by the Airbyte CDK to handle HTTP APIs. The

HTTPStream

requires a URL to perform the request and instructions on how to parse the resulting response. The

parse_response

function instructs the stream how to parse the API response. This returns an

Iterable

, whose elements are each later transformed into an

AirbyteRecordMessage

. API routes whose response contains a single record generally have a

parse_response

function that return a list of just that one response. Routes that return a list, usually have a

parse_response

function that return the received list with all elements. The

HTTPStream

must describe the schema of the records it outputs using JsonSchema. The simplest way to do this is by placing a

.json

file per stream in the

schemas

directory in the generated python module. The name of the

.json

file must match the lower snake case name of the corresponding Stream. You can also dynamically set your schema. For more information, you can refer to the [schema docs](https://docs.airbyte.com/connector-development/cdk-python/full-refresh-stream#defining-the-streams-schema). For more details, you can refer to the [HTTP Streams documentation](https://docs.airbyte.com/connector-development/cdk-python/http-streams).