@kapa.ai I'm using the redshift connector. It seems like by default, I'm retaining a copy of every sync ever run in s3? Is there a way to set this so files in s3 are automatically deleted after a period of time, say 7 days, so we only have archive for the last week?

Managing S3 Staging File Retention for Redshift The Redshift connector does have a built-in option to handle staging file cleanup, but it works at the sync level, not on a time-based retention schedule. Purge Staging Data Option The connector has a Purge Staging Data setting that, when enabled, automatically deletes the CSV staging files from S3 after each sync's

COPY

command completes. [Redshift Destination] > Normally these files are deleted after the

COPY

command completes; if you want to keep them for other purposes, set

purge_staging_data

to

false

. So if

purge_staging_data

is enabled (set to

true

), files are deleted immediately after each sync — not retained at all. If it's currently

false

(or defaulting to retaining files), that would explain why you're accumulating files. For Time-Based Retention (e.g., 7 days) The knowledge sources do not describe a built-in Airbyte mechanism to retain S3 staging files for exactly 7 days before deletion. Your options are: 1. Enable

Purge Staging Data

in the Redshift connector settings — this deletes files right after each sync (no 7-day archive). 2. Use AWS S3 Lifecycle Policies — configure an S3 lifecycle rule directly in AWS to automatically expire/delete objects after 7 days. This is an AWS-native feature and would give you the time-based retention window you're looking for, independent of Airbyte's settings. The second approach (S3 Lifecycle Policies) is likely your best path to achieving a "keep only the last 7 days" archive, since Airbyte's built-in option is all-or-nothing (delete immediately vs. keep forever). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai

purge_staging_data

is set to true. I don't think it's ever been set to false. Why do I have old files in s3?

There are a couple of known issues documented that could explain why old files are accumulating in S3 despite

purge_staging_data

being set to `true`: 1. Concurrent FlushWorker File Collision (Active Bug) There is an open bug where concurrent

FlushWorker

threads can generate S3 keys with the same epoch timestamp, causing one file to silently overwrite another. This is a race condition in the async flush infrastructure. [GitHub Issue] 2. S3 Filename Pattern Using {date} If your S3 Filename Pattern is set to

{date}

, a known issue causes duplicate files to be uploaded to the staging area. When two files share the same date-based name, the second overwrites the first — but the manifest file references both URLs, pointing to the same file. This can leave orphaned or unexpected files behind. [GitHub Issue; Forum Post] The workaround recommended by users is to switch the S3 Filename Pattern to

{timestamp}

or

{timestamp:millis}

instead of

{date}

. What to Check • S3 Filename Pattern: Switch from

{date}

to

{timestamp:millis}

if you haven't already. • Connector version: Make sure you're on the latest version of the Redshift connector, as several fixes have been shipped. That said, the knowledge sources don't provide a definitive root cause for your specific situation. If neither of the above applies, I'd recommend opening a support ticket or filing a GitHub issue with your connector version and logs for further investigation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Will anything bad happen if I delete the old files?

Based on the knowledge sources, the staging files in S3 are temporary — they are only needed during the sync process for the

COPY

command to load data into Redshift. Once that step is complete, they serve no further purpose for Airbyte. [Redshift Destination] So if these are old staging files from completed syncs, deleting them manually should be safe — your data has already been loaded into Redshift. This is also consistent with the fact that

purge_staging_data

is designed to delete them automatically after each sync with no ill effects. [Redshift Destination] However, I'd urge one caution: make sure the files you're deleting are not from a currently in-progress or recently failed sync. Deleting staging files from an incomplete sync could cause that sync to fail or result in missing data, since the

COPY

command would no longer be able to reference them. The knowledge sources don't go into further detail on edge cases around manual deletion, so if you're unsure, I'd recommend checking that all your connections show a completed (not failed or running) status before deleting. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.