Hi <@U04S8PLJWMT> (custom builder) is there a way ...
# replication-ask-ai
a
Hi @kapa.ai (custom builder) is there a way to get the child items from an incremental parent without the child to be incremental ? I have this parent that has been set as incremental and I would like for each parent synced (in incremental mode) to be able to query its children that are from another endpoint with the parent id as parameter. The children doesn't contain any timestamp so I'm not able to have it as incremental.
k
Short answer: with the current Airbyte behavior, this is not supported in the way you want. In both the Python CDK (
HttpSubStream
) and the low‑code / Connector Builder world, a child (nested) stream will re‑iterate over all parent records, not just the ones returned by the parent’s incremental window, when the child is not itself incremental. This is by design, because otherwise you could miss new child records attached to old parent records that didn’t get updated. [Nested streams; Substream behavior] Relevant points from the docs and forum: • For nested streams, the default state format is “per partition with fallback to global”, but this still assumes the child has its own incremental cursor if you want it to be incremental. [Nested streams] • Officially, if the parent is incremental and the child is not, the child will run a full refresh of the parent stream (i.e., iterate over all parent records), not just the new ones. [How sub streams work; Partitioned data behavior] • There is an open product gap specifically for “incremental parent with full‑refresh child that only uses the incremental parent slice”; users have asked for this and it’s acknowledged as not yet supported. [Incremental parent with full-refresh child; Incremental parent & full-refresh child request] Because your child endpoint has no timestamp or other cursor, Airbyte cannot make that child stream incremental in a supported way, and there is no documented mechanism in the custom/low‑code builder to say “only call the child for the parents that were just synced incrementally.” So, based on the available docs and forum answers: • You can: ◦ Make both parent and child incremental if you can define a valid cursor for the child (not your case), or ◦ Accept that the child will effectively be full‑refresh (iterate over all parents each sync). • You cannot (today, in supported/custom‑builder configuration) have: ◦ Parent = incremental ◦ Child = non‑incremental ◦ And have the child only called for the parents returned in the parent’s incremental window. If you need this behavior, the only options suggested in the community are workarounds outside the documented feature set (e.g., custom logic in a fully custom connector), which are not described in the provided knowledge sources, so I can’t give a supported recipe for that. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.