Hi. We are using airbyte 1.8 here. How to streaming data from postgres to iceberg+nessie table, but following the existing iceberg table partitioning? We pre-created the iceberg table with partitioning. Then when the sync is over, there is new folder on the object storage, containing the data, but there is no partitioning. Meanwhile if we use normal INSERT query to the iceberg table, it automatically apply the partitioning (The folder location and hierarchy level are identical)

I'm sorry, but based on the available knowledge sources, Airbyte's S3 Data Lake (Iceberg) destination connector does not currently support writing data into pre-existing Iceberg table partitions or following an existing partitioning scheme. The connector creates tables and writes data without applying partitioning, even if the target Iceberg table is pre-partitioned. This is a known limitation and has been raised by users in the community, as not having partitioning can significantly impact performance for large datasets. The current behavior is that Airbyte writes data into the table, but the data is not partitioned according to the existing Iceberg table's partitioning; instead, it is just sorted by the primary key if configured, and new folders appear in object storage without partition structure. This is different from using standard INSERT queries, which do respect Iceberg's partitioning and folder hierarchy. There is an open discussion and feature request about this limitation, where users have requested that Airbyte should support writing data into Iceberg tables using the existing partitioning (e.g., using the cursor field as a partition key and allowing configuration of partition granularity). However, as of now, this feature is not implemented, and there is no official workaround documented in the knowledge sources. The Airbyte team is aware of the need for this feature, but there is no timeline or solution provided yet. If this functionality is critical for your use case, you may want to follow or contribute to the ongoing discussion and feature requests in the Airbyte community and GitHub issues for updates and possible workarounds in the future [Create a partition to the iceberg tables created by S3-Data-Lake Connector](https://github.com/airbytehq/airbyte/discussions/57501). In summary: Airbyte 1.8 does not currently support streaming data from Postgres to a pre-partitioned Iceberg+Nessie table while preserving or following the existing partitioning scheme. Data will be written without partitioning, regardless of the pre-existing table structure.