Does the Iceberg destination support configuring a partition spec, and does it preserve partitioning on pre-created tables? I'm running destination-iceberg 0.3.52 with a Postgres CDC source, writing ~16 tables into AWS Glue catalog / S3. All tables are being created unpartitioned — every snapshot shows

changed-partition-count=1

, so the entire table is a single partition. On the largest table that's now ~70 GB across 18k files with 543M equality deletes, and it makes compaction and snapshot expiry very expensive since nothing can be scoped per-partition. Two questions: 1. Is there any way to configure a partition spec (e.g.

day(_airbyte_extracted_at)

) on tables the connector creates — in 0.3.52 or any later version? 2. If not, does the connector write into a pre-existing table and preserve its partition spec, or does it drop and recreate the table on first sync? I'm planning a full reload and could pre-create the tables with partitioning, but only if the spec survives.

Iceberg Destination: Partition Spec Support 1. Configuring a partition spec (e.g. day(_airbyte_extracted_at)) No — there is currently no way to configure a partition spec in the connector. This applies to both

destination-s3-data-lake

v0.3.52 and all versions up to the latest (v0.3.53 as of the changelog). [GitHub Issue] The issue has been filed and escalated internally. The root cause is exactly as you'd expect: •

IcebergUtil.createTable

applies a sort order on primary key fields but never calls

withPartitionSpec(...)

— tables are always created unpartitioned. •

IcebergTableSynchronizer.maybeApplySchemaChanges

evolves columns, sort order, and identifier fields only — it never modifies the partition spec. • Neither

GcsDataLakeSpecification

nor

S3DataLakeSpecification

exposes any partition configuration option. There's also a secondary blocker for

day(_airbyte_extracted_at)

specifically:

_airbyte_extracted_at

is currently stored as a

long

(epoch millis), not a

timestamptz

, so Iceberg's `day`/`month`/`year`/`hour` transforms (which require `date`/`timestamp(tz)` source types) cannot be applied to it directly. [GitHub Issue] 2. Pre-creating tables with a partition spec — will the connector preserve it? Yes, this should work as a workaround, and it's the officially acknowledged interim approach. The Airbyte team confirmed: > "Pre-creating the table with a partition spec (via Spark/Trino/PyIceberg) and then letting the connector load into it should work with the current code — since

IcebergTableWriterFactory

already dispatches to partitioned writers when

table.spec().isPartitioned()

." [GitHub Issue] The write-side infrastructure is already in place:

IcebergTableWriterFactory

branches on

table.spec().isUnpartitioned

and uses

PartitionedAppendWriter

/

PartitionedDeltaWriter

when the table is partitioned. [IcebergTableWriterFactory] The key behaviors that make this safe: •

IcebergUtil.createTable

loads an existing table without resetting its spec. •

IcebergTableSynchronizer

never modifies the partition spec during schema sync. Important caveats (from the issue): • This is unsupported — no config, no tests, no docs. • It is fragile to future connector changes: a future refactor could inadvertently reset the spec. • You'll need to manage the partition spec externally (PyIceberg, Spark, Trino) before the first sync. Summary For your full reload scenario, pre-creating the tables via PyIceberg with your desired partition spec before triggering the sync is the most viable path today. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements