@Rocky In shared data architecture, what is the data persistence format used for starrocks native tables? Is it Arrow or Parquet?

In StarRocks shared-data architecture, the data persistence format for native tables is neither Arrow nor Parquet. Instead, StarRocks uses its own internal proprietary columnar storage format, often referred to as segment files (with a

.dat

extension). This format is consistent across both the shared-nothing and shared-data (storage-compute separation) architectures. Key Characteristics of the Persistence Format: * Columnar Storage: Data is organized into segments, where each segment is a self-contained columnar file (typically around 512 MB). * Data Pages: Within a segment, data is stored in 64 KB pages. These pages are encoded (e.g., Dictionary, RLE, Delta) and compressed (defaulting to LZ4). * Rich Indexing: The format embeds multiple levels of indexing directly into the file, including: * Zone-maps: Min/max values for pages and segments to enable pruning. * Short-key (Prefix) Index: For fast range and point seeks. * Ordinal Index: To map row positions to page offsets. * Bitmap/Bloom Filter Indexes: Optional indexes stored alongside the data. * Shared-Data specifics: In the shared-data architecture, these identical segment files are stored in remote object storage (like S3) or HDFS, rather than on local BE disks. Why not Arrow or Parquet? * Parquet: While StarRocks can read/write Parquet for external tables and data lake integration, its native format is optimized for its own vectorized execution engine and includes specialized indexes (like the Short-key index) that Parquet does not support natively. * Arrow: StarRocks uses Arrow Flight SQL as a high-performance transport protocol to move data between the engine and clients (like Python or Spark) to avoid serialization overhead, but it does not use Arrow as its on-disk persistence format. For more details, you can refer to the StarRocks Architecture Documentation. References • best_practices/table_clustering.md • table_design/table_types/primary_key_table.md • unloading/arrow_flight.md • table_design/table_design.md • sql-reference/data-types/semi_structured/VARIANT.md