Hi @kapa.ai i upgraded my destination postgres to use Direct Load. My version is now 3.0.6 I am facing this error while syncing : sourceStats must be set with recordCount for state message id=1 java.lang.IllegalArgumentException: sourceStats must be set with recordCount for state message id=1

This error is coming from Airbyte’s new bulk/direct-load state handling: every state message is now expected to include

sourceStats.recordCount

(the number of records associated with that state). If it’s missing or null, you get errors like: •

sourceStats must be set with recordCount for state message id=1

• or related exceptions such as

State message does not contain id

/ NPEs when

sourceStats

is absent. [ClickHouse NPE; State id issue] From the maintainer comment: > “It is expected that a source will always populate the source stats attribute in a state message. The source stats is the count of records associated with the state message i.e. how many records the source has read that are associated with this given state message.” [ClickHouse NPE] Given that: • The problem is not specific to Postgres 3.0.6 itself, but to the state messages your source is emitting. • This often shows up when using a custom source or a source that hasn’t been updated to the newer protocol expectations. What you need to do 1. Check your source connector (especially if it’s custom or forked): ◦ For every

STATE

message it emits, ensure: ▪︎

state.sourceStats

is present, and ▪︎

state.sourceStats.recordCount

is a non-null number representing how many records were emitted since the previous state (or cumulatively, depending on your implementation, but it must be consistent). 2. Example of required fields in a state message (conceptually): ◦

AirbyteStateMessage

must include: ▪︎

type

(e.g.,

GLOBAL

or

STREAM

) ▪︎

data

/

stream

/

global

as appropriate ▪︎

sourceStats

with

recordCount

set 3. If you are not maintaining the source: ◦ Share this error and the requirement above with the source’s maintainer; they need to update the connector to populate

sourceStats.recordCount

in all emitted state messages. 4. Until the source is fixed: ◦ Incremental syncs will not work reliably because Airbyte will refuse to persist state when

sourceStats

is missing or incomplete, leading to behavior like “Airbyte lost its cursor state” and repeated re-syncing. [ClickHouse bulk state] The docs and issues do not show a Postgres-specific toggle to bypass this check; the fix must be in the source’s state emission logic. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai my source is a postgres connector at version 2.0.33

This error is coming from the destination side (the bulk/direct-load CDK used by Postgres 3.x), but it is triggered by the state messages emitted by your source. From the bulk destination requirements, destinations must rely on the source to provide state messages that include stats, and they treat those as checkpoints. [Partner destinations] Specifically, the CDK now expects: • Every state message to have

sourceStats

•

sourceStats.recordCount

must be set (number of records associated with that state) There is a GitHub issue showing that when a source emits a state without

sourceStats

, destinations using the bulk CDK fail with a similar error / NPE. [ClickHouse NPE] Another issue shows that mismatches in these stats cause state reconciliation problems and duplicate processing. [Bulk state bug] However, the provided knowledge sources do not say: • From which version the Postgres source (like your

2.0.33

) started populating

sourceStats

, or • Whether

2.0.33

is guaranteed to be compatible with the newer bulk state expectations of Postgres destination

3.0.6

. So I cannot definitively say whether

source-postgres 2.0.33

is compliant; the docs simply state the general requirement that sources must emit state and that state is used for checkpointing, but they don’t mention

sourceStats

or version compatibility details. [Protocol state; Protocol 2.0 state] Given what we do know: 1. The destination is enforcing that

sourceStats.recordCount

be present. This is consistent with the partner destination requirements and the ClickHouse issue. [Partner destinations; ClickHouse NPE] 2. Your error means the state messages from

source-postgres 2.0.33

(for this connection) do not have

sourceStats.recordCount

set. The destination then throws

IllegalArgumentException: sourceStats must be set with recordCount for state message id=1

. Because the knowledge sources don’t include a fix or compatibility matrix, I can’t point you to an exact “upgrade to source version X” answer. What they do imply is: • The correct fix is for the source to emit state messages with

sourceStats.recordCount

populated. • This is a source-side change, not something you can toggle in the Postgres destination 3.0.6 configuration. Practical next steps (based on available info): • Confirm (via logs) that the state messages coming from your Postgres source indeed lack

sourceStats

or

recordCount

– this matches the pattern described in the ClickHouse issue. [ClickHouse NPE] • If you control the source implementation (custom fork), it must be updated to populate

sourceStats.recordCount

on every emitted state. • If you are using the standard

source-postgres

image and not modifying it, the knowledge sources do not say whether

2.0.33

is expected to support this; in that case, you’ll likely need to: ◦ Involve Airbyte support / open an issue referencing this error and the bulk state requirements, or ◦ Align both source and destination to versions that are known to work together (but the exact versions are not specified in the provided docs, so I cannot recommend a specific one from this context). If you can share a snippet of the state messages from your job logs, that would help confirm that

sourceStats.recordCount

is missing, which is exactly what the destination is complaining about. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai can you explain how does the direct load method work in postgres ?

In Postgres 3.x, “Direct Load” means Airbyte writes rows straight into the final tables in your target schema, instead of first writing to raw JSON tables and then running a Typing & Deduping SQL step. Core behavior • No intermediate raw tables (by default) Each stream is written directly to a final table in your configured schema. Raw tables are not created unless you explicitly enable the legacy “Raw tables only” mode. [Direct load concept; Postgres migration] • Same final-table structure as T+D The final tables look like the typed/deduped tables you had before, but are populated directly by the destination connector instead of via a warehouse-side SQL T+D query. [Direct load concept] • Type casting happens inside the connector The Postgres destination now performs the JSON → typed-column casting itself and inserts typed rows directly, which avoids the heavy

CAST

work that T+D used to do in your warehouse. [Direct load concept] Table schema in Postgres (Direct Load) For each stream, the final table contains your data columns plus Airbyte metadata columns: [Postgres schema map] •

_airbyte_raw_id

(

VARCHAR

): unique id per record •

_airbyte_extracted_at

(

TIMESTAMPTZ

): when the record was pulled from the source •

_airbyte_meta

(

JSONB

): metadata about the record (including validation / schema-change info) •

_airbyte_generation_id

(

BIGINT

): identifier for the sync generation Data types are mapped from Airbyte types to Postgres types (e.g.,

string → VARCHAR

,

number → DECIMAL

,

object/array → JSONB

, timestamps → appropriate `TIMESTAMP`/`TIME` types). [Postgres schema map] Sync modes & visibility With Direct Load: [What will change] • Append: records are inserted directly into the target table as they are processed (they no longer “wait” in a raw table until the end of the sync). • Dedup: for most destinations, records are written to a temporary table during the sync and periodically upserted into the final table; the temp table is dropped after a successful sync. • Overwrite / refresh & remove: still use a temporary table to support no-downtime swaps, then replace the final table at the end. Raw tables compatibility If you still need raw tables: • Enable “Raw tables only” on the Postgres destination. Then: ◦ Only raw tables are written (legacy behavior), ◦ Final tables are not created for that connection. [What will change; Postgres migration] If you need both raw and final tables, you must use two destinations (one normal, one “Raw tables only”) and sync to both in parallel. [What will change; Postgres migration] Schema evolution & errors • Direct-load connectors apply schema changes via

ALTER TABLE

(or equivalent) instead of rebuilding tables via a soft reset. Some type changes can fail if historical data can’t be cast to the new type. [What will change] • Type/serialization issues are recorded in

_airbyte_meta

with error reasons (now

DESTINATION_SERIALIZATION_ERROR

instead of the old T+D

DESTINATION_TYPECAST_ERROR

). [What will change] In short: for Postgres 3.x, Direct Load means the connector itself casts and writes directly into final, typed tables with metadata columns, improving performance and eliminating persistent raw tables unless you explicitly opt into legacy raw-table mode. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Hi @kapa.ai i am getting this error after i moved to 3.x for destination connector "ERROR: column \"_airbyte_generation_id\" of relation \"logs\" does not exist\n Position: 107" what does this mean ?

This error means your sync (or a query Airbyte is running) is trying to use the

_airbyte_generation_id

column on the

logs

table, but that column does not exist yet in Postgres. With the move to Postgres destination 3.x and Direct Load, Airbyte changed the metadata columns on final tables: • Old style:

_airbyte_loaded_at

• New style:

_airbyte_meta

(JSONB) and

_airbyte_generation_id

(BIGINT) on the final tables [Postgres migration guide; Postgres schema map]. So this error typically means: • Your existing

logs

table was created by an older (pre‑3.0.0) Postgres destination and does not yet have

_airbyte_generation_id

, but • The upgraded 3.x destination is now trying to write to that table assuming the new schema (including

_airbyte_generation_id

). The migration guide explicitly says that after upgrading you may need to adjust downstream objects and that final tables now include

_airbyte_meta

and

_airbyte_generation_id

instead of

_airbyte_loaded_at

. [Postgres migration guide] What you can do (per docs): • Let Airbyte recreate the table with the new schema by: ◦ Resetting the connection and re‑syncing, or ◦ Dropping the existing

logs

table so the destination can create it fresh with

_airbyte_generation_id

and

_airbyte_meta

. • Also review any custom SQL / views that still expect the old

_airbyte_loaded_at

column and update them to the new metadata layout. [Postgres migration guide] If you still see the error after a full reset (with the table recreated by Airbyte), then something outside the documented behavior is happening; the knowledge sources don’t provide more detail beyond the schema change itself. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai i keep seeing this error r c.z.h.p.ProxyLeakTask(run):85 Connection leak detection triggered for org.postgresql.jdbc.PgConnection@7e050be1 on thread main, stack trace follows java.lang.Exception: Apparent connection leak detected at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:99) ~[HikariCP-7.0.2.jar:?]

The knowledge sources you provided don’t mention this specific HikariCP “connection leak detection triggered” error, so I can’t give an Airbyte‑specific root cause from them. I can only explain it at a high level. This log line:

c.z.h.p.ProxyLeakTask(run):85 Connection leak detection triggered for org.postgresql.jdbc.PgConnection@... on thread main, stack trace follows java.lang.Exception: Apparent connection leak detected
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:99) ~[HikariCP-7.0.2.jar:?]

means: • HikariCP handed out a JDBC connection (

PgConnection@...

) to some code. • That code never returned the connection to the pool (i.e., it didn’t close it). • After Hikari’s leak‑detection timeout, Hikari logs this stack trace to show where

getConnection()

was called but the connection was not closed. In Airbyte terms, this usually indicates that some part of the code path using the Postgres connection is not closing it properly, but the provided forum posts and issues only show timeouts and “connection is not available” errors, not this specific leak‑detection message. [Sync shows 0 records; Failed to connect external DB] Because there’s no direct mention of this leak error in the knowledge sources, I can’t point you to a known bug or configuration fix in a specific Airbyte version. To move forward, you’d typically: • Look at the full stack trace Hikari prints after this line to see which Airbyte class/method called

getConnection()

and didn’t close it. • Use that to decide whether it’s: ◦ A known bug (then upgrade Airbyte / the connector), or ◦ A custom connector / custom code path you control that needs to ensure every acquired connection is closed in all code paths (including error cases). If you can share the full stack trace around this message (with class names and line numbers), I can help you interpret which part of Airbyte is involved, but I still won’t be able to reference a specific documented fix unless it appears in the provided knowledge sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@Ishan Anilbhai Koradiya were you able to fix the error?

The knowledge sources don’t mention this exact warning for the Postgres source 3.7.2 specifically, but they do explain the underlying requirement and show a very similar problem in another connector. Airbyte’s newer destination/CDK behavior requires that every state message from the source includes

sourceStats

with a

recordCount

value: > “It is expected that a source will always populate the source stats attribute in a state message. The source stats is the count of records associated with the state message…” [ClickHouse NPE; Partner destinations] When this is missing, destinations (including Postgres 3.x) can throw errors like: •

sourceStats must be set with recordCount for state message id=1

• or NPEs when

sourceStats

is null. [ClickHouse NPE] From the docs/issues we have: • This is enforced on the destination side, but it is caused by the state messages emitted by the source. • The ClickHouse issue shows that if a source emits state without

sourceStats

, bulk destinations fail. [ClickHouse NPE] • The general protocol docs only describe state as a black box and don’t mention

sourceStats

explicitly, so there is no version matrix telling us exactly from which Postgres source version this is guaranteed to be present. [State & checkpointing] Because of that, I cannot say with certainty, from the provided docs, whether

source-postgres 3.7.2

is supposed to always set

sourceStats.recordCount

or whether you’re hitting a bug in that version. What the docs do support is: 1. The destination is correct to expect

sourceStats.recordCount

on each state message. [Partner destinations] 2. Your warning means at least one state message from the Postgres source is missing that field. Given the limitations of the knowledge sources, the best supported steps are: • Inspect the job logs to confirm that the state messages from the Postgres source lack

sourceStats

or

recordCount

(this matches the pattern in the ClickHouse issue). [ClickHouse NPE] • If you are not running a custom fork of the source, this likely needs to be raised as a bug against

source-postgres

3.7.2, referencing: ◦ The error text, and ◦ The requirement that sources populate

sourceStats.recordCount

for each state. [Partner destinations] The provided docs do not contain a configuration toggle or workaround on the destination side to bypass this check, so any real fix has to be in the source’s state emission logic. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai sorry, the source is v3.7.2 and destination is v3.0.7

You’re hitting a known class of issues with the new bulk/CDK v3 destinations: they now require enriched state messages from the source, and your

source-postgres:3.7.2

is emitting a state without the required stats. From the destination requirements: • Destinations must treat state as checkpoints and • Must append record counts to the source’s state message before emitting it. [Partner destinations] For this to work, the source must provide

sourceStats.recordCount

on every state message. When it doesn’t, bulk destinations (like Postgres 3.x) fail with errors such as: >

sourceStats must be set with recordCount for state message id=1

This exact pattern is documented for another destination (ClickHouse): a custom source emitted state without

sourceStats

, and the destination threw an error / NPE because

sourceStats

was missing. [ClickHouse NPE] So, in your case: • Destination:

destination-postgres:3.0.7

(Direct Load, bulk CDK) • Source:

source-postgres:3.7.2

• Meaning of the warning: At least one state message from

source-postgres

is missing

sourceStats.recordCount

, which the destination now requires. The provided docs do not say: • From which

source-postgres

version

sourceStats.recordCount

is guaranteed to be set, or • Whether

3.7.2

has a known bug around this. So I can’t point to a specific “upgrade/downgrade to version X” fix from the knowledge sources. What you can do, based on the available information: 1. Confirm the missing stats in logs Inspect the sync logs and look for the raw state messages; you should see that the state with

id=1

has no

sourceStats

or no

recordCount

. This matches the ClickHouse example. [ClickHouse NPE] 2. If you’re using a custom/forked source Update it so that every emitted state message includes: ◦

sourceStats

◦

sourceStats.recordCount

= number of records associated with that state. 1. If you’re using the standard

source-postgres:3.7.2

image The docs don’t provide a workaround or config flag on

destination-postgres

to bypass this check. Since the requirement is that sources populate

sourceStats.recordCount

, this likely needs to be treated as a bug report against

source-postgres

3.7.2, referencing: ◦ The error text, and ◦ The documented requirement that destinations rely on record counts in state messages. [Partner destinations] Because the knowledge sources don’t include a compatibility matrix or a specific fix for this version pair, I can’t recommend an exact version combination as a guaranteed solution; the core, documented requirement is that the source must emit state with

sourceStats.recordCount

, and your error indicates that is not happening. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.