Hi all, what are the current thoughts on shared-data mode vs iceberg tables? mostly concerned with query performance

Hi Liran, I think the shared data mode is significantly better than using Iceberg tables. One of the most important points for me is the possibility of real-time data ingestion. Maintenance is also easier than with Iceberg. In terms of performance, I think that for some use cases, you can certainly get close to Starrock's shared data mode. However, in addition to pure partitioning, Starrock offers other features that provide significant performance advantages, especially for join-heavy queries. That's why we're currently considering switching from Iceberg to Starrock.

OK thanks a lot @Johann Brandl, may I ask about your use case for a bit? how much data etc?

Hi Liran, we use Iceberg and Dremio as a Data-Lake for Reporting and Analysis for some of our systems (MES and other platforms). Most of the data comes from Oracle databases and is ingested via Debezium CDC to Kafka and then via Spark to Iceberg. The data should actually be about 20 TB but growing. We encounter serious performance problems with iceberg. The main reason for this is not iceberg itself but poor table design. I tried the same with starrocks (routine load from kafka). What I particularly like about Starrocks: - Ingest is significantly easier (routine load or Kafka connector with Upsert) - Performance: - Data ingest is significantly faster than with Iceberg (no separate jobs for insert and compaction) - Searching for primary keys is significantly faster than with Iceberg -Joins are significantly faster than with Iceberg - Data transformations can be performed directly during ingest (routine load and Kafka connector) (in my case, Double -> BIGINT, Unixtime -> Date/Datetime) - We wouldn't need a separate query engine with Starrocks (Dremio) Regarding performance, I compared our databases (Oracle, Iceberg, Starrocks) with a few sample queries. Unfortunately, the comparison isn't reliable, as I have different data volumes in the systems. Furthermore, Iceberg itself isn't optimized. Nevertheless, Starrocks was better than Oracle and much better than Iceberg for my use cases.

I'll offer an opposite take - I'm currently considering moving away from using proprietary starrocks tables with shared data in favor of open table formats so I can more easily use spark for geospacial processing that starrocks is lacking.