When doing output to S3, if the connected faces retries, the parquet files from multiple attempts can leave side by side. We would like to identify which files belong to which attempt. We currently do not see a way to mark the files this way, we've assumed it should be possible with the placeholders like

{timestamp}

or

{sync_id}

but couldn't find one for "attempt". Any ideas?

Based on the available knowledge sources, there is no placeholder variable specifically for "attempt number" or "attempt ID" in the S3 path format or filename pattern configuration. The currently documented path format variables are: [S3 setup guide] •

${NAMESPACE}

,

${STREAM_NAME}

•

${YEAR}

,

${MONTH}

,

${DAY}

,

${HOUR}

,

${MINUTE}

,

${SECOND}

,

${MILLISECOND}

,

${EPOCH}

•

${UUID}

And the supported filename pattern placeholders are:

{date}

,

{date:yyyy_MM}

,

{timestamp}

,

{timestamp:millis}

,

{timestamp:micros}

,

{part_number}

,

{sync_id}

,

{format_extension}

. None of these correspond to an attempt number or attempt ID. What you can infer indirectly: •

{sync_id}

in the filename pattern identifies the sync, but not the specific attempt within that sync. •

${UUID}

or

${EPOCH}

would make filenames unique across attempts, but wouldn't let you directly correlate a file to a specific attempt number. The underlying issue (files from failed attempts persisting alongside successful ones) is a known limitation that has been raised by the community. [Forum discussion] The suggested workaround at the time was to track files written per attempt and clean up on failure, but this was not natively supported. Unfortunately, the knowledge sources do not contain information about an attempt-level placeholder being added. If this is a critical requirement, I'd recommend checking the Airbyte GitHub issues/discussions or opening a feature request for an

${ATTEMPT_NUMBER}

or similar path variable. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I cannot read this forum discussion, what does it say?

The forum discussion [Duplicated data on S3] is about a user experiencing duplicate data in their S3 destination after syncs that succeed following multiple failed attempts. Here's a summary of what it says: • Setup: The user is syncing from MS SQL Server to S3 (destination-s3 v0.3.5) on Kubernetes. • Problem: When a sync succeeds after one or more failed attempts, data from the failed attempts is not cleaned up in the S3 bucket. This results in duplicate files/data sitting alongside the data from the successful attempt. • Root cause of failures: The user suspected MS SQL Server timeouts during large table syncs, causing generic validation errors. • User's proposed solution: Keep track of files written during each attempt and delete them if the attempt fails — but this was not natively supported. • Community response: A community member acknowledged it looked like a bug and suggested creating a GitHub issue. Another suggestion was to use custom dbt for deduplication, but the user noted S3 doesn't support dbt transformations and their tables lacked a suitable dedup field. • Resolution: No fix was provided in the thread; the user was directed to open a GitHub issue. This confirms that there is no built-in mechanism in the S3 destination to clean up files from failed attempts, and no attempt-level identifier in the filename pattern to distinguish them. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

what is the title of the discussion exactly? I'll google it with this @kapa.ai

The exact title of the forum discussion is: "Duplicated data on s3 destination after multiple failed attempts" You can find it at: https://discuss.airbyte.io/t/1398 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai how am I expected to track which files were written during which attempt?

Based on the knowledge sources available, there is no documented native mechanism in Airbyte's S3 destination to track which files were written during which specific attempt. As noted in the community discussion [Duplicated data on S3], a user explicitly proposed that the solution should be: > "keeping track of the files written during an attempt, and then deleting them if the attempt fails" — but this was acknowledged as something not natively supported at the time. What the S3 destination does provide is a generation ID tag on each S3 object (

x-amz-meta-ab-generation-id

), which identifies the generation (i.e., a full refresh cycle), not the individual attempt. [S3 sync modes] The knowledge sources do not contain enough information to describe a supported, documented way to correlate S3 files to a specific attempt number. I'd recommend: 1. Checking the Airbyte GitHub for any updates or open issues on this topic. 2. Opening a feature request for attempt-level tagging or a cleanup mechanism for failed attempts. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.