@Rocky are their any benchmarks for shared-data vs shared-nothing architecture for query performance and ingestion throughput(ingesting through parquet files/ arrow stream/arrow files)

@Rocky are their any benchmarks for shared-data vs shared-nothing architecture for query performance and ingestion throughput(ingesting through parquet files/ arrow stream/arrow files)

Yes, there are official benchmarks and performance reports comparing shared-data (storage-compute separation) and shared-nothing (storage-compute coupled) architectures in StarRocks. 1. Query Performance Benchmark (SSB 1TB) StarRocks documentation provides a direct comparison of query latency using the Star Schema Benchmark (SSB) 1TB dataset on StarRocks v3.3.0. Architecture | Total Latency (SUM of 13 queries) Shared-data (with Data Cache) | 83,444 ms (Fastest) Shared-nothing | 100,406 ms Shared-data (no Data Cache) | 238,412 ms Conclusion: With Data Cache enabled, the shared-data architecture was 1.2x faster than shared-nothing in this test. This is attributed to optimizations like Parallel Scan and I/O merge that effectively hide object storage latency. Without cache, shared-nothing is significantly faster due to local disk I/O. ────────── 2. Ingestion Throughput While a single side-by-side table comparing the two architectures for ingestion is less common, StarRocks has published specific benchmarks for shared-data ingestion throughput and optimizations: * Shared-Data Optimization: In StarRocks v3.2.3, a benchmark for high-concurrency real-time ingestion (simulating 1MB file stream loads) showed that enabling

lake_enable_batch_publish_version

improved throughput from 14.5 MB/s to 90.8 MB/s (a ~6x increase) at high concurrency (256 threads). * Parquet/Bulk Ingestion (StarRocks 3.5+): * Small-File Problem: Shared-data architectures can suffer from "small-file" overhead during bulk Parquet/Arrow ingestion because every flush creates a remote object. * Load Spill: StarRocks 3.5 introduced a Load Spill mechanism specifically for shared-data mode. It spills data to local disk first to perform a "centralized merge" before writing to object storage, significantly increasing write throughput for massive historical datasets. * Arrow Stream Ingestion: * Arrow Flight SQL: Starting from v3.5.1, StarRocks supports Arrow Flight SQL, which allows for end-to-end columnar data transfer. This eliminates row-to-column serialization overhead, providing "zero-copy" high throughput for data science and ML workloads. Summary of Differences Metric | Shared-Nothing | Shared-Data (with Cache/3.5+) Query Latency | Optimized for low-latency, real-time queries via local disk. | Comparable or better than shared-nothing when data is cached. Ingestion Throughput | Highly stable; limited by local disk IO and CPU. | Can reach higher aggregate throughput using object storage bandwidth, but requires batch optimizations (v3.5+) to avoid small-file overhead. Scaling | Rebalancing required (data movement) when scaling. | Instant scaling; no data movement required. For the most up-to-date performance figures, especially regarding the newer Load Spill and Arrow Flight features, refer to the StarRocks 3.5 Release Notes and the Shared-data Feature Support pages). References • deployment/shared_data/feature-support-shared-data.md • knowledge/usage/load/shared_data_batch_ingestion.md • faq/shared_data_faq.md • quick_start/shared-data.md

@Mayank Virmani docs.starrocks.io/docs/benchmarking/TPC_DS_Benchmark This one too