Environment: • Airbyte v1.8.5 (being upgraded to 2.1) • Postgres → S3 destination:

definitionId 4816b78f

(legacy S3 destination, v1.9.8) — writes Airbyte metadata columns (

_airbyte_meta

,

_airbyte_raw_id

,

_airbyte_extracted_at

,

_airbyte_generation_id

) directly into Parquet files as data columns • S3 → Snowflake: Source S3 v4.15.4 with

delivery_type: use_records_transfer

+ Snowflake destination v4.x Problem: The legacy S3 destination embeds Airbyte metadata columns into the Parquet files. When the S3 source reads these files using

use_records_transfer

mode, it passes all columns — including the embedded metadata — to the Snowflake destination v4.x. The Snowflake destination then tries to add its own

_AIRBYTE_META

column when creating the table, causing:

SQL compilation error: duplicate column name '_AIRBYTE_META'

Proposed fix being evaluated: Change

delivery_type

from

use_records_transfer

to

use_file_transfer

in the S3 source configuration, which bypasses record deserialization and passes Parquet files directly to Snowflake without adding metadata columns.

Analysis of Your Duplicate _AIRBYTE_META Column Issue Your diagnosis is correct. The root cause is well-understood: the legacy S3 destination embeds Airbyte metadata columns (

_airbyte_meta

,

_airbyte_raw_id

, etc.) as data columns in the Parquet files, and when Snowflake destination v4.x reads those records, it attempts to add its own reserved metadata columns — resulting in the duplicate column name SQL compilation error. What the Knowledge Sources Confirm There is a known bug in the Bulk CDK's

ColumnNameResolver

where user columns whose names collide with Airbyte-reserved names (

_airbyte_raw_id

,

_airbyte_extracted_at

,

_airbyte_meta

,

_airbyte_generation_id

) were not detected as conflicts, causing duplicate column names in

CREATE TABLE

DDL — crashing destinations like Snowflake. [CDK fix PR] The fix (pre-seeding the processed column names set with

Meta.COLUMN_NAMES

) was released in

bulk-cdk-core-load

v1.0.14, and the Snowflake destination picked it up in v4.0.44. [Snowflake CDK bump] > If your Snowflake destination is already on v4.0.44 or later, user columns named

_airbyte_meta

etc. will be auto-renamed to

_airbyte_meta_1

instead of crashing. This may be the cleanest fix without changing your pipeline architecture. Evaluating Your Proposed Fix (use_file_transfer) The knowledge sources confirm that

use_file_transfer

copies raw files without parsing their contents — Airbyte passes the Parquet bits directly to the destination without deserializing records or adding metadata columns. [Delivery methods] However, there are important caveats the knowledge sources highlight: •

use_file_transfer

is only supported on specific file-based connectors and requires the destination to also support file transfers. The only destination listed as supporting file transfers is S3 — not Snowflake. [File transfers] • This means switching S3 source to

use_file_transfer

mode with a Snowflake destination may not be a supported configuration. Recommended Path The knowledge sources do not contain enough detail about your specific

use_records_transfer

vs

use_file_transfer

behavior with Snowflake as a destination to fully validate the proposed fix. I'd strongly recommend verifying whether

use_file_transfer

is even a valid option for an S3→Snowflake pipeline before pursuing it, and checking whether upgrading the Snowflake destination to v4.0.44+ resolves the collision directly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

based on this log, what woud you recommend me. You can ignore previous suggetions @kapa.ai

Sorry @Passarelli, I encountered an error processing the file(s) you uploaded: Text file is too large (max 60K characters)

2026-06-11 114638 info ----- START CHECK ----- 2026-06-11 114638 info 2026-06-11 114643 info Connector exited, processing output 2026-06-11 114643 info Output file jobOutput.json found 2026-06-11 114643 info Connector exited with exit code 0 2026-06-11 114643 info Reading messages from protocol version 0.2.0 2026-06-11 114643 info INFO main i.m.c.e.DefaultEnvironment(init):168 Established active environments: [k8s, cloud, cli, destination, connector] 2026-06-11 114643 info INFO main i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 Setting log level 'ERROR' for logger: 'net.snowflake' 2026-06-11 114643 info INFO main i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 Setting log level 'ERROR' for logger: 'com.zaxxer.hikari.pool' 2026-06-11 114643 info INFO main i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 Setting log level 'ERROR' for logger: 'com.zaxxer.hikari' 2026-06-11 114643 info INFO main i.a.c.AirbyteConnectorRunnable(run):35 Executing class io.airbyte.cdk.load.check.CheckOperation operation. 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 SELECT COUNT(*) > 0 AS SCHEMA_EXISTS FROM "AIRBYTE_DATABASE".INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = ? 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 CREATE OR REPLACE TABLE "AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222" ( "_AIRBYTE_RAW_ID" VARCHAR NOT NULL, "_AIRBYTE_EXTRACTED_AT" TIMESTAMP_TZ NOT NULL, "_AIRBYTE_META" VARIANT NOT NULL, "_AIRBYTE_GENERATION_ID" NUMBER, "TEST_KEY" VARCHAR NOT NULL ) 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 CREATE STAGE IF NOT EXISTS "AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."airbyte_stage__AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222" 2026-06-11 114643 info INFO main i.a.i.d.s.w.l.SnowflakeInsertBuffer(flush):82 Beginning insert into "AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222"... 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 PUT 'file:///tmp/snowflake1221936242533479724.csv.gz' '@"AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."airbyte_stage__AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222"' AUTO_COMPRESS = FALSE SOURCE_COMPRESSION = GZIP OVERWRITE = TRUE 2026-06-11 114643 info INFO main i.a.i.d.s.w.l.SnowflakeInsertBuffer(flush):87 Copying staging data into "AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222"... 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 COPY INTO "AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222"("_AIRBYTE_RAW_ID", "_AIRBYTE_EXTRACTED_AT", "_AIRBYTE_META", "_AIRBYTE_GENERATION_ID", "TEST_KEY") FROM '@"AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."airbyte_stage__AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222"' FILE_FORMAT = ( TYPE = 'CSV' COMPRESSION = GZIP FIELD_DELIMITER = ',' RECORD_DELIMITER = ' ' FIELD_OPTIONALLY_ENCLOSED_BY = '"' TRIM_SPACE = TRUE ERROR_ON_COLUMN_COUNT_MISMATCH = FALSE REPLACE_INVALID_CHARACTERS = TRUE ESCAPE = NONE ESCAPE_UNENCLOSED_FIELD = NONE ) ON_ERROR = 'ABORT_STATEMENT' PURGE = TRUE files = ('snowflake1221936242533479724.csv.gz') 2026-06-11 114643 info INFO main i.a.i.d.s.w.l.SnowflakeInsertBuffer(flush):94 Finished insert of 1 row(s) into "AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222". 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 SELECT COUNT(*) AS TOTAL FROM "AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222" 2026-06-11 114643 info INFO main i.a.i.d.s.s.SnowflakeDirectLoadSqlGeneratorKt(andLog):44 DROP TABLE IF EXISTS "AIRBYTE_DATABASE"."AIRBYTE_SCHEMA"."_AIRBYTE_CONNECTION_TEST_8881E7FEFDD04313B382E622BF1FF222" 2026-06-11 114643 info INFO main i.a.c.AirbyteConnectorRunnable(run):47 Flushing output consumer prior to shutdown. 2026-06-11 114643 info INFO main i.a.c.AirbyteConnectorRunnable(run):49 Completed integration: airbyte/destination-snowflake. 2026-06-11 114643 info Checking for optional control message... 2026-06-11 114643 info Writing output of 71c2208f-58b7-4dd9-bfef-fca2a9f4f606_1152_0_check to the doc store 2026-06-11 114643 info Marking workload 71c2208f-58b7-4dd9-bfef-fca2a9f4f606_1152_0_check as successful 2026-06-11 114643 info Deliberately exiting process with code 0. 2026-06-11 114643 info 2026-06-11 114643 info ----- END CHECK ----- 2026-06-11 114643 info 2026-06-11 114644 info APPLY Stage: BUILD — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info APPLY Stage: CLAIM — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info Claimed: true for workload 2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync via API in dataplane AUTOc278275f-92a9-4058-a6b4-e67c72c57593 (8aa56571-8a1a-434a-a01a-52ad748f465d) 2026-06-11 114644 info APPLY Stage: LOAD_SHED — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info APPLY Stage: CHECK_STATUS — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info No pod found running for workload 2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync 2026-06-11 114644 info APPLY Stage: MUTEX — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info Mutex key: 2c785a09-5f1c-493f-b5e5-53f97ae83a17 specified for workload: 2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync. Attempting to delete existing pods... 2026-06-11 114644 info Existing pods for mutex key: 2c785a09-5f1c-493f-b5e5-53f97ae83a17 deleted. 2026-06-11 114644 info APPLY Stage: ARCHITECTURE — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info APPLY Stage: LAUNCH — (workloadId=2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync) 2026-06-11 114644 info [initContainer] image: airbyte/workload-init-container:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={}) 2026-06-11 114644 info Launching replication pod: replication-job-1152-attempt-0 (selectors = {}) with containers: 2026-06-11 114644 info [source] image: airbyte/source-s3:4.15.4 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.5}, additionalProperties={}) 2026-06-11 114644 info [destination] image: airbyte/destination-snowflake:4.0.43 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.5}, additionalProperties={}) 2026-06-11 114644 info [orchestrator] image: airbyte/container-orchestrator:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={}) 2026-06-11 114653 info Attempting to update workload: 2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync to LAUNCHED. 2026-06-11 114653 info Pipeline completed for workload: 2c785a09-5f1c-493f-b5e5-53f97ae83a17_1152_0_sync. 2026-06-11 114700 info Running replication worker... 2026-06-11 114700 info Starting replication worker. job id: 1152 attempt: 0 2026-06-11 114701 info 2026-06-11 114701 info ----- START REPLICATION ----- 2026-06-11 114701 info 2026-06-11 114701 info Running destination... 2026-06-11 114701 info Writing messages to protocol version 0.2.0 2026-06-11 114701 info Reading messages from protocol version 0.2.0 2026-06-11 114701 info Reading messages from protocol... @kapa.ai

Log Analysis The logs confirm your environment details from the previous conversation. Key observations: ✅ Connection Check Passed (v4.0.43) The

CHECK

phase completed successfully — Snowflake destination v4.0.43 can connect, create tables, stage, copy, and drop test tables without issue. ⚠️ The Critical Version Detail Your sync is running:

[destination] image: airbyte/destination-snowflake:4.0.43

This is one version behind the fix. The

_airbyte_meta

/ reserved column name collision fix was released in v4.0.44, which bumped the Bulk CDK from

1.0.13

→

1.0.14

. [Snowflake CDK bump] The fix in v4.0.44 pre-seeds the collision detection set with all reserved Airbyte column names, so a user column named

_airbyte_meta

(coming from your legacy S3 Parquet files) would be auto-renamed to

_airbyte_meta_1

instead of crashing with a duplicate column error. [CDK fix PR] What the Logs Show About the Sync Start The replication pod launched successfully with: •

source-s3:4.15.4

•

destination-snowflake:4.0.43

The log cuts off at

Reading messages from protocol...

— this is where the actual record flow (and the duplicate column crash) would occur. Recommendation Upgrade

destination-snowflake

to v4.0.44 or later. This is the most direct fix for your duplicate

_AIRBYTE_META

column error, requiring no pipeline architecture changes. [Snowflake CDK bump] The knowledge sources do not contain enough information to confirm whether v4.0.44 is available on your Airbyte v1.8.5 instance or whether it requires upgrading to Airbyte v2.x first — you may need to verify connector version availability in your deployment. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai seems like 4.0.43 is the latested version for this snowflake connector https://docs.airbyte.com/integrations/destinations/snowflake

The knowledge sources do not contain the current Snowflake destination changelog page content, so I cannot independently verify what the latest published version is. However, based on what is in the knowledge sources: • PR #79181 explicitly bumps

destination-snowflake

from 4.0.43 → 4.0.44 and updates

docs/integrations/destinations/snowflake.md

with a changelog entry. [Snowflake CDK bump] • That PR was merged and contains the reserved column name collision fix. It's possible that v4.0.44 exists in the codebase but hasn't yet been rolled out to your Airbyte Cloud/self-managed instance — the PR itself notes an active progressive rollout warning for

destination-snowflake

, meaning the version may be deployed gradually rather than immediately to all users. [Snowflake CDK bump] What you can do in the meantime Since upgrading to v4.0.44 may not yet be available to you, the most practical workarounds are: 1. Pre-process your Parquet files to strip the embedded

_airbyte_*

metadata columns before they reach the S3 source → Snowflake pipeline. 2. Rename the conflicting columns at the S3 source level using stream field remapping if your Airbyte version supports it. 3. Monitor the rollout — once v4.0.44 reaches your instance, the fix will auto-rename colliding columns (e.g.

_airbyte_meta

→

_airbyte_meta_1

) without any pipeline changes. I'd recommend checking Airbyte Community Slack or opening a support ticket to confirm the rollout timeline for v4.0.44 on your specific deployment. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.