@kapa.ai, im running a sync with child and parent streams. When testing in the builder UI, my child stream is succesful. When running the sync, the parent stream hangs on

Read 66619 records from parent stream

. I see no attempt to try child streaming or anything else after this point in the logs. How can I pinpoint the issue

This is a known challenge with parent/child streams in Airbyte. Here are some steps to help you pinpoint the issue: 1. Check the Sync Logs in Detail Navigate to your connection's Job History tab, click the three grey dots next to the stuck sync, and select View logs. Search for terms like

Error

,

Exception

, or

Fail

to identify any underlying errors. If the in-app viewer isn't enough, download the logs to search across all attempts. [Browsing logs] 2. Enable Parent Stream Caching A very likely cause of the hang is that when the child stream starts, it re-reads the entire parent stream from scratch via HTTP requests. With 66,619 parent records, this means tens of thousands of individual API calls for the child stream partitions, which can cause the sync to appear stuck or run out of memory. To avoid this, enable caching on the parent stream so its records are written to a temporary file on disk and reused by the child stream without re-fetching:

class YourParentStream(HttpStream):
    ...

    @property
    def use_cache(self) -> bool:
        return True

class YourChildStream(HttpSubStream):
    ...

[Nested Streams & Caching] 3. Understand the Root Behavior The

HttpSubStream

class generates one stream slice per parent record. With ~66k parent records, the child stream will attempt ~66k individual API requests one at a time. This is by design but can cause: • Memory exhaustion • Very long run times that look like a hang • The process appearing stuck at the parent read stage before child slicing even begins [Forum: parent streams] 4. If Using the Low-Code Builder If you built this connector in the Connector Builder UI (not Python CDK), caching is handled differently and there have been known bugs with incremental parent/child stream state not being respected correctly. In that case, reviewing your YAML configuration and checking for known issues may be necessary. [Github Issue] ------------------------- Summary: The most likely cause is the sheer volume of parent records (66k+) causing the child stream to stall while generating slices or making individual API calls. Enabling

use_cache = True

on the parent stream is the recommended first fix if you're using the Python CDK. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai how do I enable use_cache in YAML format for the parent

In the YAML-based (low-code) connector configuration, caching for parent streams is controlled via the

use_cache

property on the

HttpRequester

component. According to the documentation, parent streams automatically have caching enabled by default. However, you can explicitly set it:

requester:
  type: HttpRequester
  url_base: "https://api.example.com"
  use_cache: true

The docs note: [HttpRequester reference] > "Parent streams automatically have caching enabled. Only set this to false if you are certain that caching should be disabled, as it may negatively impact performance when the same data is needed multiple times." So if your parent stream is already defined as a parent in a

ParentStreamConfig

, caching should already be active by default. If you're still seeing the hang with 66k+ records, the issue may not be caching-related but rather the sheer volume of child stream slices being generated — one per parent record — which can make the sync appear stuck even when it's technically still running. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Cache is enabled by default and the stream was left running overnight, with no sign of progress. What else can be done to do this more efficiently

Given that caching is already enabled and the sync ran overnight without progress, the core problem is almost certainly the volume of parent records (66k+) generating one API call per record for the child stream. Here are the options supported by the knowledge sources: 1. Use

incremental_dependency

(High Impact) If your parent stream supports incremental sync, enable

incremental_dependency

on the substream partition router. This allows the parent stream to be read incrementally rather than in full, so only recently updated parent records are enumerated for child slices. Requirement: The API must update the parent record's cursor whenever child records are added or updated. If this isn't guaranteed by the API, child records on older parents could be missed. Example state structure this produces:

{
  "parent_state": {
    "parent_stream": {
      "timestamp": "2024-08-01T00:00:00"
    }
  },
  "states": [
    {
      "partition": "A",
      "timestamp": "2024-08-01T00:00:00"
    },
    {
      "partition": "B",
      "timestamp": "2024-08-01T01:00:00"
    }
  ]
}

[Incremental Syncs - Nested Streams] 2. Use

global_substream_cursor

(For Very Large Parent Streams) If your parent stream has an extremely high number of partitions (like your 66k records), you can use a single global cursor for all partitions. This significantly reduces state size and avoids managing per-partition state. Caveats: • The global cursor is only updated at the end of the sync • If the sync fails, only the parent state is updated (if

incremental_dependency

is also enabled) • Use cautiously — it sacrifices per-partition granularity [Incremental Syncs - Nested Streams] 3. Add a

record_filter

on the Parent Stream You can add a

record_filter

directly in your YAML to pre-filter parent records before child slices are generated, reducing the number of child API calls made:

record_selector:
  type: RecordSelector
  extractor:
    type: DpathExtractor
    field_path: [...]
  record_filter:
    type: RecordFilter
    condition: "{{ record['updated_at'] > stream_state['updated_at'] }}"

Caveat: This filter is not exposed in the Connector Builder UI — you must edit the YAML directly. Switching back to the UI after adding this may cause the filter to be deleted. [Forum: Pre-filtering parent stream] ------------------------- Bottom line: The most impactful change for your situation would be enabling

incremental_dependency

so the parent stream is read incrementally rather than all 66k records on every sync. The

global_substream_cursor

is a secondary option if partition state management itself becomes the bottleneck. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai this is an initial sync where everything is being sent to the destionation, so incremental_dependency is unlikely to help. What other options do I have

You're right that

incremental_dependency

won't help on an initial full sync. Based on the knowledge sources available, the options are limited for this specific scenario. Here's what is documented: 1. Use

global_substream_cursor

This is the most relevant option for a large initial sync. Instead of managing state per partition (one per parent record), it uses a single global cursor, which reduces overhead significantly when the parent stream has a very high number of partitions (like your 66k records). Set

global_substream_cursor: true

on your `DatetimeBasedCursor`:

incremental_sync:
  type: DatetimeBasedCursor
  global_substream_cursor: true
  # ... other cursor fields

[DatetimeBasedCursor reference] 2. Use a

Grouping Partition Router

If your API supports filtering by multiple IDs in a single request, the Grouping Partition Router can batch multiple parent partitions together into one API call, dramatically reducing the number of requests:

partition_router:
  type: GroupingPartitionRouter
  group_size: 50  # batch 50 parent IDs per request
  underlying_partition_router:
    type: SubstreamPartitionRouter
    # ... your existing parent stream config

Note: Per-partition incremental syncs may not work as expected with grouping. [Partition Router Types] ------------------------- Beyond these two options, the knowledge sources don't provide additional documented solutions for optimizing an initial full sync with a large parent stream. The fundamental challenge — one child API call per parent record — is a known limitation of the parent/child pattern at scale, as noted in the forum discussion. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.