Hey team! 🙂 I have a quick question regarding batch size in Airbyte. I deployed a custom HTTP source connector which queries large amounts of data. With the apparently hard-coded batch_size of 10K records, my machine (32GB RAM) runs out of memory. How can I adjust the batch size? I saw that this was recently added for JDBC-based connectors. Is an equivalent available for the Python cdk? Thank you!

There is no hard coded batch size for python sources. So you'd need to modify your code to add a notion of page size.

I see, thank you for your reply, @Sherif Nada. Do you have any hint regarding what would be the best way to implement that? Using stream_slices?

how does the underlying API enable you to set batch size? is it a query param?

The API does not have a query param to set batch size. The connector iterates over pages. (How) could I configure the connector such that it saves the records after each page/X pages?

How big are the pages? It’s curious that the RAM runs out. If it’s just a “vanilla” API whose responses are not very big (a page is no more than say 5mb) then I’d guess the issue is not the source connector?

The Content-Length of a page's multipart output can be > 20.000.000

huh…it feels like even then that shouldn’t account for a ton of memory usage? How many of these are you running at a time?

Depending on the number of pages it can be a total of >200GB

yes but is the connector saving those in memory? the only relevant number here should be how many bytes are used in memory at any given time, not the full size of the dataset

Hi. when you pull that much data you need to stream it and process in batches -before you are sending it to airbyte streamers - I would suggest adding dask for this type of loading and if needed add parallel workload for that to achieve faster result for each batch size that you are sending aslso - what format is the response? josn? csv? etc..?