@kapa.ai If I have a source from SQL Server + CDC and i push the data into S3. the cdc holds only part of the history, so regular tables will be used as well on the first pull. How should i expect the data to be looked like on the first run and the ones after? CDC - like or regular table?

On the first run and later runs you’ll see both behaviors, but in different phases of the sync. How Airbyte behaves with CDC sources (like SQL Server) 1. First run (initial snapshot) ◦ Airbyte does a full snapshot of the current state of the regular tables using

SELECT

(this is effectively a Full Refresh). ◦ This snapshot is not coming from the CDC log; it’s just the current state of the table at that moment. [CDC 1.7] ◦ After that snapshot, Airbyte starts reading from the CDC log position it recorded. So on the first run, your S3 data will look like: ◦ A full copy of the table as it exists at snapshot time (regular-table-like view). ◦ Plus any changes that happened after the snapshot, coming from CDC logs (CDC-like records with

_ab_cdc_*

metadata). 2. Subsequent runs ◦ Airbyte no longer re-reads the whole table. ◦ It reads only changes from the CDC log since the last stored log position. [CDC 1.7] ◦ These records include CDC metadata columns such as

_ab_cdc_lsn

,

_ab_cdc_updated_at

,

_ab_cdc_deleted_at

. [CDC 1.7] So on later runs, your S3 data is CDC-like only (append of change events). What you should expect in S3 • Initial run: ◦ Files that effectively represent a snapshot of the table (regular-table-like state at that time). ◦ Plus CDC-style change records for any modifications that occurred after the snapshot completed. • Later runs: ◦ Only CDC-style change records (INSERT/UPDATE/DELETE) with

_ab_cdc_*

metadata. [CDC 1.7] If you then load this S3 data into a warehouse with “incremental deduped + history” or similar, Airbyte’s normalization can build both a current (deduped) table and a history (SCD) table from those CDC-style records. [CDC tutorial] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

OK, can i move the files that are being created in the S3 bucket to Success / Failure subfolders so i can manage a el from there?

The knowledge sources don’t explicitly describe a built‑in way for the S3 destination to move files into

success/

or

failure/

subfolders after a sync, so anything like that would need to be handled outside Airbyte (for example, by your own post‑processing job or workflow). What is configurable in the S3 destination is: • S3 Bucket Path – the base “folder” under the bucket where Airbyte writes data. • S3 Path Format – a template for how data is organized under that bucket path (e.g.

${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_

). [S3 setup] • S3 Filename pattern – controls the file name itself using placeholders like

{date}

,

{timestamp}

,

{part_number}

,

{sync_id}

,

{format_extension}

. [S3 setup] These settings let you control where and how Airbyte writes files, but the docs do not mention any mechanism to: • Move already‑written files into a different folder based on success/failure, or • Automatically clean up or re‑organize files from failed attempts (in fact, a forum post notes that failed‑attempt data can remain in S3). [S3 duplication forum] So: • You can choose a path convention that encodes status or run ID (e.g. include

{sync_id}

in the path or filename), but • You cannot configure Airbyte itself (per the docs) to move files into

Success/

or

Failure/

folders after the fact; that would require an external process watching the bucket and moving/tagging objects based on your own logic. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I don't want airbyte to do that. I want to do it when i pull the s3 files to other location, with a different tool, so i know which files I've managed to copy and which not

You can absolutely do that with your own tooling; Airbyte doesn’t constrain how you post‑process the S3 files. What Airbyte guarantees (and what you can rely on for your EL step) is: • Files are written under:

<bucket-name>/<S3 Bucket Path>/<S3 Path Format><partition-id>.<format-extension>

where

S3 Bucket Path

and

S3 Path Format

are fully configurable. [S3 setup] • Default

S3 Path Format

is:

${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_

and you can customize it with variables like

${NAMESPACE}

,

${STREAM_NAME}

,

${YEAR}

,

${MONTH}

,

${DAY}

,

${EPOCH}

,

${UUID}

, etc. [S3 path format] • You can also control the file name via S3 Filename pattern, using placeholders such as

{date}

,

{timestamp}

,

{part_number}

,

{sync_id}

,

{format_extension}

. [S3 filename pattern] Given that, a common pattern for your external tool is: 1. Use

S3 Path Format

and/or

S3 Filename pattern

to encode run identity, e.g. include

{sync_id}

or a timestamp in the path or file name. 2. Your external process: ◦ Lists files for a given run (e.g. by prefix or

sync_id

). ◦ Copies them to your target. ◦ On success, moves/renames them into a

success/

prefix; on failure, into

failure/

. Airbyte itself doesn’t track which individual files you’ve copied with another tool, but its naming/path conventions are stable and configurable enough for you to implement that tracking externally. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.