Tip for other Snowflake Users: I've been using the "Snowflake" Destination for ages, and have found it to be quite slow depending on the source... also had issues with the COPY INTO trimming spaces in columns. I've just switch to using the "S3 Data Lake" Destination with the newly released Polaris/Snowflake Open Catalog Support. I've seen upto 6x performance increase, especially when the source is Oracle/Postgres/MySQL. The set up is • an "Internal" Catalog in a Snowflake Open Catalog Account • Create a catalog integration in your main snowflake account, connecting to your polaris catalog • Create a catalog-linked database from the integration so it's always upto date Useful links: • This guide is great, I mainly followed this but replaced the spark bits with Airbyte setup https://docs.snowflake.com/en/user-guide/opencatalog/tutorials/open-catalog-gs#use-case-2-sync-apache-iceberg-tables-from-snowflake-to-open-catalog • Creating a catalog linked database https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database • Don't forget to set

ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS

on your integration.... https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration Hope this helps reduce your Snowflake bill, your Airbyte infra bill, and also gets you some hands-on exp with Snowflake x Iceberg tables ❄️

Interesting. Out of curiosity, had you used the latest version of the Snowflake connector with direct loading (version 4.0+)?

Yeah - we've been using 4.0.7 recently, but I wouldn't say it's been any slower than previous versions. I was pretty surprised by the performance increase, considering the S3 Data Lake option is, I assume, writing the data to iceberg tables without any sort of query/MPP engine.

this is very promising! i'd been looking to test Iceberg tables and this may just be the reason to do so

There's an initial overhead for sure (a second or two) which seems roughly the same no matter the table size, but I've found that's also the case with Databricks... so all in all, not much difference!

nice, thanks for letting me know!

One thing to note - when you set up a catalog linked database, it doesn't automatically set

AUTO_REFRESH

on the iceberg tables as default, despite the catalog syncing on an interval, so make sure you do that bit, otherwise you'll be wondering why Airbyte has synced but your data isn't fresh 😅

hah, will keep that in mind!

@Roberto Tolosa I was actually wrong about that last note, it was just good ol' Snowflake permissions giving me the runaround.

Thx for sharing.