Hi All I have a question about Flink + Iceberg and partitioning... well, on best practise when it comes to partitioning in general. So I come to those more knowledgeable than I. What the the relationship between a Flink PK (for upsert / merges) and iceberg partitioning? It seems that if you try to partition that table it MUST be by the Primary Key column? But if that's the case, and there are a high cardinality of keys, then each partition would have a very small amount of records (often only 1), which would be counter productive? I could understand if we could use the bucket partition function, but it's not yet supported. What is best practise in cases like these? If there is a link / docs, I could be pointed at? (I've also asked on the Iceberg community...) Thanks in advance

I think you're touching exactly on why FLIP-376 was created https://cwiki.apache.org/confluence/display/FLINK/FLIP-376%3A+Add+DISTRIBUTED+BY+clause 🙂

Thanks, I'll take a read of that. But for my current use case, I expect we'd need to go unpartitioned, and use other levers (like bloom filters and metadata) to manage performance?

I think so yes

Many thanks, good sir! At least I know I'm not 100% insane 🙂 .