Using CDC for postgres, _airbyte_extracted_at woul...
# replication-ask-ai
y
Using CDC for postgres, _airbyte_extracted_at would be updated if there is no updates. But what field can I use to capture that Airbyte scanned the source for updates? Essentially I'm trying to build a process to catch un-refreshed sources
✅ 1
k
Based on the knowledge sources available, I can partially address your question, but there are some limitations to what I can confirm. What the metadata fields tell you For CDC syncs, the key Airbyte metadata fields are: •
_airbyte_extracted_at
: Timestamp for when Airbyte pulled the event from the data source. This reflects when a record change was captured, not when a scan/sync occurred. •
_airbyte_meta
: Contains a
sync_id
field that monotonically increases across syncs. This is the most relevant field for your use case — even if no records changed, the
sync_id
within
_airbyte_meta
increments with each sync run. [Airbyte metadata fields] Using sync_id to detect scans The
sync_id
inside
_airbyte_meta
increases monotonically across syncs, regardless of whether records were updated. You could query the maximum
sync_id
across your destination table to determine the last sync that touched those records:
Copy code
SELECT MAX((_airbyte_meta->>'sync_id')::int) AS latest_sync_id
FROM your_table;
However, this only tells you the latest sync that wrote records. If a CDC sync ran but found no changes, no new records would be written, so
sync_id
in the destination wouldn't advance for that table. Important caveat The knowledge sources do not explicitly describe a field that is updated on every sync scan regardless of whether records changed. The
_airbyte_loaded_at
field exists in raw tables (for destinations using typing and deduping), but: • It reflects when records were loaded into the destination, not when a scan occurred. • It is not guaranteed to be present in all destinations or in final tables. [Airbyte metadata fields] Recommendation For your use case of detecting "un-refreshed sources," the knowledge sources don't confirm a built-in field that tracks every sync attempt (including no-op scans). You may need to rely on Airbyte's sync logs or job history via the Airbyte API rather than destination table metadata, as the destination only reflects records that were actually written. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.