Antoine Balliet
06/28/2023, 1:29 PMkapa.ai
06/28/2023, 1:29 PMkapa.ai
06/28/2023, 1:30 PMstream_slices function in Airbyte's connector development kit (CDK) allows for finer grain control of when state is checkpointed. Conceptually, a Stream Slice is a subset of the records in a stream which represent the smallest unit of data which can be re-synced. The stream_slices function generates a Slice for every request to be made. For example, if an API is able to dispense data hourly, the stream_slices function can return a list of 24 Slices, each with a different hourly timestamp to be used when creating request. If the stream fails halfway through (at the 12th slice), then the next time it starts reading, it will read from the beginning of the 12th slice. In your case, you can generate one stream slice per batch of ids. The path method will be called for each stream_slice. You can then process the stream_slice value in the path function to build the URL you want. For more information, you can refer to the [Stream Slices guide](https://github.com/airbytehq/airbyte/tree/8500fef4133d3d06e16e8b600d65ebf2c58afefd/docs/connector-development/cdk-python/stream-slices.md) and the [Incremental Streams guide](https://docs.airbyte.com/connector-development/cdk-python/incremental-stream) in the Airbyte documentation. Please note that the actual implementation will depend on the specific API you are working with and its support for batch requests.