Hello community, I reported an issue regarding the Connector Builder / Declarative CDK a while ago, and I would heavily appreciate it if someone from the team could take a look, as it severely impacts sync performance. When using

incremental_dependency: true

on parent-child streams, the child stream drops its cursor context after the first sync, forcing the parent stream into an unexpected full refresh loop on subsequent syncs (making them 100x–1000x slower). I just updated the issue with a comprehensive root-cause analysis (assisted by GitHub Copilot and verified against the latest Concurrent CDK source code). I also provided: 1. A functional connector-level workaround using custom components (StateMigration) that anyone can apply today. 2. The exact lines in the CDK core (

substream_partition_router.py

) where the context is being lost. Here is the link to the issue: github.com/airbytehq/airbyte/issues/61567 Could you please help review the proposed architectural fix or provide recommendations to get this merged? Thanks!

Thank you for the detailed write-up and for following up on this issue. Based on the knowledge sources available, here's what I can share: Summary of the Known Issue The behavior you're describing — where

incremental_dependency: true

on parent-child streams causes the parent to re-execute from

config.start_date

on subsequent syncs — is a known and documented behavior in the Airbyte declarative CDK. [SubstreamPartitionRouter discussion] The root cause is that when

SubstreamPartitionRouter

re-executes the parent stream to generate partition IDs, it does not carry over the persisted cursor state by default, causing it to start from

config.start_date

every sync. [SubstreamPartitionRouter discussion] What incremental_dependency: true Is Supposed to Do Setting

incremental_dependency: true

on the parent stream config inside the substream's partition router is the intended mechanism to fix this. With it enabled: • The parent re-execution inside the router uses the persisted parent-cursor state, so subsequent syncs request data from

<last_high_watermark>

instead of

<config.start_date>

. • The parent state is persisted under a

parent_state

key in the emitted state messages.

sales_details:
  type: DeclarativeStream
  retriever:
    type: SimpleRetriever
    partition_router:
      type: SubstreamPartitionRouter
      parent_stream_configs:
        - stream: "#/definitions/streams/sales"
          parent_key: id
          partition_field: sale_id
          incremental_dependency: true   # <-- the key flag

[SubstreamPartitionRouter discussion] Important Prerequisite (and a Known Limitation)

incremental_dependency: true

only works correctly if: 1. The child stream has its own

incremental_sync

block. If the child has no own cursor, it gets a

FinalStateCursor

which never persists

parent_state

, making the flag silently inert. [docs PR clarification] 2. The parent record's cursor field is updated whenever any child record changes. If this API-level guarantee is not met, child records added to older parent records will be missed. [Incremental Syncs docs] Related Open Issues Your issue is not isolated. Similar problems have been reported for: • source-gitlab: Child stream

merge_request_discussions

with

incremental_dependency: true

always gets full refreshes because the platform doesn't pass the full input state to the connector. [GitLab issue] • source-calendly:

event_invitees

substream always fully syncs due to the same

incremental_dependency

limitation. [Calendly issue] Recommendations 1. Verify the inert-flag case: Confirm your child stream has its own

incremental_sync

block — without it,

incremental_dependency: true

has no effect at runtime. [docs PR clarification] 2. Inspect emitted state: After a sync, check whether

parent_state

appears in the state output. If it's absent, the flag is not being activated. 3. Consider

global_substream_cursor: true

as an alternative if per-partition granularity isn't critical — it uses a single global cursor and can reduce unnecessary re-fetches. [Incremental Syncs docs] Regarding Your Proposed Fix The knowledge sources do not contain enough information about the specific architectural fix or

StateMigration

workaround you've proposed in the issue. For the fastest path to review, I'd recommend: • Tagging the issue with

area/connector-builder

and

type/bug

on GitHub. • Referencing the related issues (#50962) to demonstrate the broader impact. • Joining the Airbyte Community Slack and posting in the

#connector-development

channel to get direct attention from the CDK team. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai and did you reviewed all the issue and my last comments?

I'm sorry, but I have to be transparent with you: I do not have access to the specific issue you linked (#61567) or your latest comments on it. It was not included in the knowledge sources provided to me for this conversation. My previous response was based on related but separate issues and documentation about

incremental_dependency: true

and

SubstreamPartitionRouter

— not on your specific issue or your root-cause analysis. To get a proper review of your issue and proposed fix, I'd recommend: 1. GitHub — Make sure the issue is clearly labeled (e.g.,

area/connector-builder

,

type/bug

) so the right team sees it. 2. Airbyte Community Slack — Post in the #connector-development channel with a direct link to the issue and a summary of your root-cause analysis. 3. If you're an Airbyte Cloud customer — Open a support ticket referencing the issue URL for a guaranteed response. I apologize for any confusion my previous response may have caused by not making this limitation clear upfront. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.