@kapa.ai how the parameter stream_reponse works with the HttpRequester? I am trying to find solutions to avoid OOM issues that are happening with pods running one specific integration I have

stream_response in HttpRequester and Avoiding OOM How stream_response Works The

stream_response

parameter on

HttpRequester

controls whether the underlying HTTP request is made in streaming mode (i.e.,

stream=True

is passed to

requests.Session.send()

). • When

stream_response=False

(default): the entire response body is downloaded and buffered in memory before any records are processed. For large responses (multi-GB), this can cause OOM kills. • When `stream_response=True`:

session.send()

returns immediately after receiving headers, and the response body flows incrementally — records are processed as bytes arrive, keeping memory usage bounded. In the declarative CDK,

stream_response

is typically derived from the decoder. For example, in `BingAdsReportDownloadRequester`:

def __post_init__(self, parameters: Mapping[str, Any]) -> None:
    if self.decoder and hasattr(self.decoder, "is_stream_response"):
        self.stream_response = self.decoder.is_stream_response()
    super().__post_init__(parameters)

[Bing Ads components.py] In the Python CDK (legacy

HttpStream

), the equivalent is overriding

request_kwargs()

to return

{"stream": True}

. Common OOM Root Causes From real connector issues, OOM typically happens due to: 1. No streaming:

session.send()

buffers the full response via

b"".join(chunks)

— ~2x the payload size in memory. 2. No read timeout: If the server stalls mid-transfer,

ssl.read()

blocks forever. 3. In-memory decoders: Decoders that load the entire decompressed document + parsed object tree (e.g., a naive

GzipJsonDecoder

) can require 4–5x the uncompressed size. [Marketo issue] Solutions by Connector Type 1. Python CDK (HttpStream) — Override request_kwargs

def request_kwargs(self, **kwargs) -> Mapping[str, Any]:
    return {"stream": True, "timeout": (30, 300)}

This was the fix applied to `source-marketo`:

stream=True

prevents full buffering, and

timeout=(30, 300)

(30s connect, 300s read) prevents indefinite hangs on stalled connections. [Marketo PR] [Marketo streaming PR] The

source-iterable

connector does the same for large NDJSON exports:

def request_kwargs(self, stream_state, stream_slice=None, next_page_token=None):
    return {
        **super().request_kwargs(stream_state, stream_slice, next_page_token),
        "stream": True,
    }

def parse_response(self, response, **kwargs):
    for obj in response.iter_lines():  # iterate line-by-line, not all at once
        record = json.loads(obj)
        yield record

[Iterable streams.py] 2. Declarative CDK — Use a Streaming Decoder Set

stream_response=True

on the

HttpRequester

and pair it with a decoder that reads the response incrementally (e.g., using

response.raw

). The Bing Ads connector uses a custom

BingAdsReportZipCsvDecoder

that spools to disk and parses CSV row-by-row:

def is_stream_response(self) -> bool:
    return True

def decode(self, response):
    spool = tempfile.SpooledTemporaryFile(max_size=5 * 1024 * 1024)
    raw = response.raw
    # stream chunks to disk, then parse CSV row-by-row
    ...
    for row in reader:
        yield row

[Bing Ads decoder] 3. Large JSON Reports — Use JsonItemsDecoder / Streaming JSON For gzip-compressed JSON arrays, the fix is to use a streaming JSON parser (like

ijson

) that yields one array element at a time. This was the fix for

source-amazon-seller-partner

Brand Analytics reports, reducing peak memory from ~15 GB to ~300 MB. [Amazon SP PR] [Amazon SP PR 2] 4. High Concurrency — Cap Concurrent Workers If OOM is caused by many concurrent streams each buffering data, cap the concurrency level.

source-s3

was reduced from 100 to 20 concurrent file readers to stay within a 2 GiB container limit. [S3 PR] Summary Checklist If you share more details about your specific integration (connector type, response format, error logs), I can give more targeted advice. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements