Hey team! With the low-code CDK, what happens when the partition_router is an incremental stream? Do we need to do something specific on the child stream?

Hi Elliot, I’m not sure I quite follow are you talking about substreams + incremental? On the declarative stream component, we now have an

incremental_sync

field used to declare how cursor based incremental syncs will function. The

partition_router

on a retriever is used for iterating over a set of static or dynamic values

Hi Brian, I’m in Elliot’s team, following up on this: We’re not sure of what’s going on with a child stream whose parent is an incremental stream - Does the child sync only run on the filtered parent records ? Is the child sync automatically declared a incremental sync ?

Does the child sync only run on the filtered parent records ? Is the child sync automatically declared a incremental sync ?

The child sync is not automatically declared an incremental syncs. You can have a parent stream that is incremental, but the child stream is full refresh. In that scenario, even if a parent stream is incremental, during the child stream sync, we would use all the parent elements as inputs to the child stream to sync all the substream records

Makes sense thanks !

How to use only the records incrementally extracted for the child stream? Is it possible?

hmmm that I’m not certain of, that might be a bit of a more complex workflow, but is that a must have for the API you are trying to implement? although it might result in slightly more API calls to the parent stream, as long as the child is incremental, that stream’s records should only contain ones from after the child cursor state

Yes unfortunately having the parent stream not incremental results in too many API calls 😕 Thanks for your answer

This is now supported in the Connector Builder UI

Hey @Lake (Airbyte) ! Thanks for the update, what version of the manifest would this be ? and what version of the airbyte cdk would it be compatible with ?

It is available in manifest and airbyte CDK version 1.4.0, and this Connector Builder UI feature is now available in Airbyte Cloud. If you use OSS, the UI feature will be available in the next OSS release