@kapa.ai which is the right method I need. I am sendign a query with a row limit of 100. If I read 100 rows, I want to send the next request. I dont have server side pagination. Which is the method I need from the python-cdk?

Based on your description, it sounds like you need to implement a custom pagination method using the Airbyte Python CDK. The most appropriate approach for your scenario would be to use the

next_page_token

method in combination with

request_params

. Here's how you can implement this: 1. In your stream class, override the

next_page_token

method. This method should return a token (in your case, it could be the next offset) when there are more pages to fetch, or

None

when pagination is complete. 2. Use the

request_params

method to include the pagination parameters in your API requests. Here's an example of how you might implement this:

python def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:     # Assuming your response contains the total number of records     records = response.json()     if len(records) == 100:  # If we received the full 100 records         # Get the current offset from the request parameters         current_offset = int(response.request.params.get('offset', 0))         # Return the next offset         return {'offset': current_offset + 100}     else:         # If we received fewer than 100 records, we're done paginating         return None  def request_params(     self, stream_state: Mapping[str, Any], stream_slice: Mapping[str, Any] = None, next_page_token: Mapping[str, Any] = None ) -> MutableMapping[str, Any]:     params = {'limit': 100}     if next_page_token:         params.update(next_page_token)     return params

This approach is described in the Airbyte documentation on [HTTP streams and pagination](https://docs.airbyte.com/connector-development/cdk-python/http-streams#pagination). Remember, you'll need to adjust this code to fit your specific API's response structure and pagination mechanism. [Airbyte CDK Python HTTP Streams documentation](https://docs.airbyte.com/connector-development/cdk-python/http-streams#pagination)

@kapa.ai but I dont get the offset from my api. I just need to know if we reached 100 records than send the next request

A message from kapa.ai

In your case, you can keep track of the number of records processed in your connector and use that to determine when to send the next request. Here's how you can do it: 1. Override the

read_records

method in your stream class. This method is responsible for making the API requests and processing the responses. 2. In this method, keep a count of the records processed. When this count reaches 100, reset it and make the next API request. Here's an example of how you might implement this:

python class MyStream(HttpStream):     ...     def read_records(         self,          sync_mode: SyncMode,          cursor_field: List[str] = None,          stream_slice: Mapping[str, Any] = None,          stream_state: Mapping[str, Any] = None     ) -> Iterable[Mapping[str, Any]]:         record_count = 0         while True:             response = self._send_request()             records = response.json()             if not records:                 break             for record in records:                 yield record                 record_count += 1                 if record_count == 100:                     break             if record_count < 100:                 break             record_count = 0

In this example,

_send_request()

is a placeholder for the method you use to send requests to your API. You'll need to replace it with your actual request method. This approach is based on the Airbyte documentation on [how the Airbyte CDK read command works](https://github.com/airbytehq/airbyte/discussions/33814).