Hi folks 👋 I'm running some research on Flink + Iceberg use cases, so for any iceberg users here your input will be highly appreciated. 1. How do u use Iceberg along with Flink? i.e do you use it to get data from Kafka into the datalake and then add also other query engines on-top? 2. Do you use it directly for CDC data? i.e CDC DB data -> data lake? 3. After you offload into the data lake are you doing extra processing with Flink + Iceberg? i.e create more tables in a streaming fashion? 4. What issues / painpoints do you come across? Overall any use cases you could share will be highly appreciated. Thanks 😄

We're using Iceberg for storing logs via Kafka. We're also using Flink for querying from Iceberg, but that has proven pretty slow. We're looking into where the slowness is coming from, and it seems a lot of it is on building the query itself rather than the query execution.

@Jad Naous partitionioning, merging small files (and making sure they are sized good), zorder sorting might help there