@kapa.ai, how to store partition by iceberg tables on s3 using s3 data lake destination

Iceberg Partitioning with S3 Data Lake Destination Based on the knowledge sources, the current state of the S3 Data Lake destination connector does not natively support configuring an Iceberg partition spec. Here's what you need to know: Current Behavior • The connector creates unpartitioned Iceberg tables by default. No

PartitionSpec

is set during table creation in

IcebergUtil.createTable

. [GitHub Issue] • The

S3DataLakeSpecification

configuration exposes no partition setting. [S3 Spec] • Interestingly, the write side already supports partitioned tables —

IcebergTableWriterFactory

dispatches to `PartitionedAppendWriter`/`PartitionedDeltaWriter` when

table.spec().isPartitioned()

is true. The gap is solely that no partition spec is ever assigned at table creation. [GitHub Issue] Workaround: Pre-create the Table with a Partition Spec You can pre-create the Iceberg table externally (e.g., via Spark, Trino, or PyIceberg) with your desired partition spec, and then let the Airbyte connector write into it. This works because: 1.

IcebergUtil.createTable

will

loadTable

if the table already exists, without resetting its spec. 2. The schema synchronizer never modifies the partition spec. 3. The writer factory will automatically use the partitioned writers when it detects

table.spec().isPartitioned()

. > ⚠️ This workaround is unsupported, undocumented, and fragile to future connector changes. Use with caution. [GitHub Issue] Feature Request Status A GitHub issue has been opened requesting first-class partition spec configuration support for both

destination-s3-data-lake

and

destination-gcs-data-lake

. The proposed feature would allow declaring

(column, transform)

pairs (e.g.,

day

,

month

,

year

,

hour

,

bucket[N]

,

truncate[W]

,

identity

) in the connector config. This has been escalated internally but is not yet implemented. [GitHub Issue] Recommendation • Short-term: Pre-create your Iceberg table with the desired partition spec using PyIceberg or another tool, then sync with Airbyte. • Long-term: Follow or upvote the GitHub issue #81351 to track when native partition spec configuration is added. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.