@kapa.ai getting this error on posthog connector as source and clickhouse as destination java.lang.IllegalArgumentException: sourceStats must be set with recordCount for state message id=1

Thinking completed This error originates from the destination connector (ClickHouse v2, which uses the Bulk CDK) during state reconciliation. Cause of the Error In newer Airbyte destination connectors built on the Bulk CDK (such as ClickHouse v2), the destination validates checkpoint state messages before committing them [Github Issue (open): [destination-clickhouse v2] Failing With IllegalStateException]. Specifically, the

StateStore

requires that every incoming state message contain a

sourceStats

object with a valid `recordCount`:

val stats =
    requireNotNull(msg.sourceStats) {
        "sourceStats must be set with recordCount for state message id=${key.id}"
    }
histogramStore.acceptExpectedCounts(key, stats.recordCount)

[StateStore.kt] The destination uses

sourceStats.recordCount

to verify that all records emitted by the source for that checkpoint batch have arrived and been flushed before committing the state [State coordination in socket mode]. If the source (in this case, the PostHog connector) emits an

AirbyteStateMessage

where

sourceStats

is missing or null, the destination fails immediately with:

java.lang.IllegalArgumentException: sourceStats must be set with recordCount for state message id=1

[StateStore.kt; Github Issue (open): [destination-clickhouse] Sync fails with error java.lang.NullPointerException]. How to Resolve The knowledge sources do not contain connector-specific release notes for PostHog or ClickHouse detailing which exact version introduced or resolved this incompatibility. However, based on how Airbyte handles this protocol requirement: 1. Update the PostHog Source Connector: Older connector versions or custom implementations may not emit the

sourceStats

attribute on state messages [Github Issue (open): [destination-clickhouse] Sync fails with error java.lang.NullPointerException]. Upgrading the PostHog source to the latest available version ensures it complies with current Airbyte protocol standards by attaching

sourceStats

with

recordCount

. 2. Update or Downgrade Destination ClickHouse: If updating PostHog is not possible, check if an update is available for

destination-clickhouse

, or test an earlier version of ClickHouse that does not strictly enforce

sourceStats

validation in the CDK reconciler [Github Issue (open): [destination-clickhouse] Sync fails with error java.lang.NullPointerException]. 3. Report to Airbyte: If both connectors are already on their latest official versions, the PostHog connector has a bug where it emits state messages without the required

sourceStats

. In that case, consider reporting this incompatibility on the Airbyte GitHub Issues repository so the connector can be updated to include

sourceStats

. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.