@Rocky we want to write and read from iceberg variant datatype, how mature is StarRocks engine regarding it (newest versions), read/write, is there any known bugs that are not suitable for production right now

StarRocks support for the Iceberg VARIANT data type was introduced in v4.1. As this is a very recent release (v4.1 debuted in mid-2026), it is considered a "new" feature. Here is the maturity breakdown for production considerations: 1. Read/Write Capabilities * Read Support: StarRocks can query

VARIANT

columns from Iceberg tables stored in Parquet format. It provides a suite of functions (e.g.,

get_variant_int

,

variant_query

,

variant_typeof

) to extract and navigate nested semi-structured data. * Write Support: You can write

VARIANT

data to Iceberg tables. StarRocks supports casting JSON strings or complex SQL types (ARRAY, MAP, STRUCT) into

VARIANT

for sinking into Iceberg. * Vectorized Execution: v4.1 integrates the Iceberg Variant format directly into its vectorized execution engine. This allows the engine to perform field extraction and filtering on the binary-encoded data without expensive JSON parsing at runtime. 2. Known Limitations (Current Version) While functional, there are several limitations that impact production design: * Unshredded Only: Currently, StarRocks only supports unshredded variant values for both read and write. If your existing Iceberg data is stored using "shredded" Parquet columns (a performance optimization in the Parquet Variant spec), StarRocks may only be able to access the fallback unshredded binary. * 16 MB Size Limit: Individual

VARIANT

values are limited to 16 MB. * Catalog Specific: The

VARIANT

type is only supported in the Iceberg Catalog. It is not yet supported for StarRocks native (Internal) tables. * Parquet Only: Support is restricted to Iceberg tables using the Parquet file format with variant encoding. 3. Production Maturity & Risks * New Feature: Since this was introduced in v4.1, it has not yet undergone years of battle-testing in diverse production environments compared to standard types. * Bugs: There are no widely reported "showstopper" bugs as of the latest release notes, but the "unshredded only" limitation is the most likely source of interoperability issues with other engines (like Spark 4.0+) that might default to shredded writes. * Recommendation: For production, it is suitable for "Schema-on-Read" workloads where you need to avoid rigid DDL changes, provided your individual records stay under 16MB and you are using StarRocks v4.1+. If performance for specific nested fields is critical, consider that the lack of "shredding" support means StarRocks still has to read the full variant binary for extraction, though it is still faster than raw JSON strings. Relevant Documentation Reference: * StarRocks v4.1 VARIANT Data Type References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • quick_start/iceberg.md • sql-reference/data-types/semi_structured/VARIANT.md

@Rocky does shredded support is coming soon feature? what is the performance difference between STRUCT/TEXT/SHREDDED VARIANT/NON-SHREDDED VARIANT (for rows of 10KB~)

Support for the Iceberg VARIANT data type in StarRocks was introduced in v4.1. While it is a major step forward for semi-structured data, it is currently in its first phase of maturity. 1. Is Shredded Support "Coming Soon"? Yes. Shredded variant support is a key target on the 2026 StarRocks Roadmap (GitHub #67632) as part of full Iceberg V3 specification compatibility. * Current Status (v4.1): Supports unshredded (binary blob) read/write. It can navigate the binary structure using offsets but still reads the entire blob for the column. * Near Future: The engineering team is working on "adaptive shredding," which will allow StarRocks to automatically treat frequently accessed sub-fields in the Variant as separate columns, similar to how it handles "Flat JSON" for internal tables. ────────── 2. Performance Comparison (~10KB rows) For a 10KB row size, the performance bottleneck shifts from metadata handling to I/O and CPU parsing costs. Here is how the formats compare when querying a specific nested field: Format | Relative Performance | Why? STRUCT | 1x (Fastest) | Truly columnar. StarRocks only reads the specific sub-column needed. Zero parsing overhead at query time. SHREDDED VARIANT | ~1.1x - 1.5x | Performance is very close to

STRUCT

for shredded fields because the engine reads the extracted sub-column directly. NON-SHREDDED VARIANT | ~3x - 10x Slower | Better than text because it uses a binary format (offset-based navigation), but the engine must still load the full 10KB blob for every row to find the field. TEXT (JSON STRING) | ~20x - 50x Slower | Worst performance. Every row requires full text parsing (lexing/regex) to find a single field, which is extremely CPU-intensive for 10KB records. ────────── 3. Known Issues & Production Readiness * Interoperability: If you write data from StarRocks v4.1, it will be unshredded. If you use other engines (like Spark 4.0+ or Snowflake) to read this, they will fall back to binary navigation, losing their "shredding" speed advantage. * Internal Table Support: The

VARIANT

type currently only works for Iceberg Catalog. If you need semi-structured data in StarRocks' native storage, you must use the

JSON

type with Flat JSON enabled (which provides shredding-like performance automatically). * Size Limit: Individual Variant values are capped at 16MB. * Production Verdict: * Suitable for: Log analytics, event streams, and "Schema-on-Read" where flexibility is more important than absolute peak performance. * Avoid for: High-concurrency, low-latency "point lookups" or massive aggregations on a single nested field (use

STRUCT

or StarRocks native tables with Flat JSON for that). Authoritative Source: StarRocks v4.1 VARIANT Documentation References • sql-reference/data-types/semi_structured/VARIANT.md • using_starrocks/Flat_json.md • sql-reference/data-types/semi_structured/JSON.md

@Rocky if in starocks the column is of type json and in iceberg its struct/variant. if we use starRocks also as a federated query engine can we access data in starrocks and iceberg with arrow notation or dot notation ?

In StarRocks, the syntax for accessing nested fields depends on the specific data type of the column. While there isn't a single universal operator that covers all three (JSON, STRUCT, and VARIANT) in the same way, you can achieve federated access by understanding the following notation rules: 1. StarRocks Internal Table (

JSON

type) For native StarRocks tables using the

JSON

data type, you primarily use Arrow Notation: * Arrow Notation (

->

):

json_col -> '$.path.to.field'

or

json_col -> 'field'

. * Arrow Notation with Type Casting: You can chain arrows or use functions like

get_json_string(json_col, '$.path')

. * Dot Notation: Not supported for the

JSON

type; it is reserved for

STRUCT

. 2. Iceberg External Table (

STRUCT

type) For Iceberg tables where the schema is explicitly defined as a

STRUCT

, StarRocks uses standard SQL Dot Notation: * Dot Notation (

.

):

struct_col.field_name

or

struct_col.nested_struct.field

. * Example:

SELECT user_profile.name FROM iceberg_catalog.db.users;

* Arrow Notation: Not supported for

STRUCT

. 3. Iceberg External Table (

VARIANT

type) The

VARIANT

type (introduced in v4.1) is a binary-encoded semi-structured type. Currently, it relies on Functional Access rather than operators: * Functions:

variant_query(variant_col, '$.path')

(returns another Variant) or

get_variant_int(variant_col, '$.path')

(returns a specific type). * Unified Access Strategy: To use Arrow notation on Iceberg

VARIANT

data, you must cast it to

JSON

first:

sql
    -- Casting to JSON to use Arrow Notation
    SELECT CAST(variant_col AS JSON) -> '$.user.id'
    FROM iceberg_catalog.db.table;

Summary Table for Federated Queries If you are writing a query that joins a native StarRocks table (JSON) and an Iceberg table (STRUCT/VARIANT), here is the syntax you will use: Source/Type | Syntax | Example Internal (JSON) | Arrow (

->

) |

json_col->'key'

Iceberg (STRUCT) | Dot (

.

) |

struct_col.key

Iceberg (VARIANT) | Functions |

variant_query(v_col, '$.key')

Pro-Tip: Making it "Look" the Same If you want a unified "dot notation" experience for your end-users across both systems, you can create Views. 1. For StarRocks

JSON

columns, create a view that extracts fields into virtual columns. 2. For Iceberg

VARIANT

columns, create a view that casts them to

STRUCT

(if the schema is somewhat stable) or extracts fields via functions. Production Note: Using

CAST(variant_col AS JSON)

allows the use of Arrow notation but may incur a small parsing overhead compared to using the specialized

get_variant_*

functions which are optimized for the binary Variant format. References • sql-reference/data-types/semi_structured/VARIANT.md • sql-reference/sql-functions/json-functions/json-query-and-processing-functions/arrow-function.md • sql-reference/data-types/semi_structured/STRUCT.md References • sql-reference/data-types/semi_structured/VARIANT.md • sql-reference/sql-functions/json-functions/json-query-and-processing-functions/arrow-function.md • sql-reference/sql-functions/json-functions/overview-of-json-functions-and-operators.md

@Rocky • Mode 1 (Classic Shared-Nothing): Ideal if you already run StarRocks Backends (BEs) with local SSDs. Partitions younger than 7 days live on local SSD. When a partition crosses the 7-day threshold, StarRocks automatically writes the segment files to S3 and clears local disk space. • Mode 2 (Cloud-Native Shared-Data): Ideal for cloud deployments. All data resides on S3 immediately upon write. Compute Nodes (CNs) use local SSDs as a Lakehouse Data Cache. Data accessed within 7 days remains hot in local NVMe cache, while older data is pulled from S3 on demand. If we want this approach, starrocks only, without Iceberg, hot in local ssd and cold in s3 starrocks format - is it production ready and stable?

Yes, both modes are production-ready and stable, provided you choose the one that aligns with your infrastructure (On-Prem vs. Cloud). Since you are looking for a "StarRocks-only" approach using the native StarRocks storage format (segment files), here is the stability and maturity breakdown: Mode 1: Classic Shared-Nothing (Hierarchical Storage) This is the most mature "hot/cold" approach in StarRocks, having been in production for several years (since the v2.x era). * How it works: You define two storage paths on your BE (e.g., local SSD and an S3-backed FUSE mount or StarRocks' native S3 cool storage integration). You set a

storage_cooldown_time

or

storage_cooldown_second

at the partition or table level. * Maturity: Highly Stable. This is the standard for users who want to manage their own hardware but offload historical data to cheaper storage. * Production Suitability: Excellent for on-premises or fixed-size clusters. * Known Limitation: Scaling out requires rebalancing data across BEs, which can be time-consuming for very large datasets. Mode 2: Cloud-Native Shared-Data (Architecture of Choice) Introduced in v3.0 and significantly hardened in v3.1+, this is the modern architecture of StarRocks. Most new large-scale cloud deployments (AWS/GCP/Azure) use this mode. * How it works: StarRocks writes data directly to S3. The local SSD on the Compute Nodes (CN) acts as a transparent cache. You control "hot" data using

datacache.partition_retention_days

or by cache size. * Maturity: Production Ready. It is the primary architecture for StarRocks' own managed cloud service and is used by many Fortune 500 companies for petabyte-scale analytics. * Production Suitability: Best for cloud environments where you need to scale compute (CNs) up and down instantly without moving data. * Known Limitation: * As of current versions, it does not support the native

BACKUP

and

RESTORE

commands (you typically rely on S3's native versioning/replication). * Querying "cold" data (data not in cache) will have higher latency as it must be pulled from S3. Comparison for Your Use Case Feature | Mode 1 (Shared-Nothing) | Mode 2 (Shared-Data) Storage Format | StarRocks Native (Segment) | StarRocks Native (Segment) Stability | Mature (Years in Prod) | Stable (Proven since v3.1) Hot/Cold Logic | Time-based movement (SSD -> S3) | LRU Cache-based (Hot on SSD) Scaling | Slow (requires data rebalance) | Instant (Stateless compute) Reliability | Multi-replica on local disks | S3 durability (11 nines) Recommendation * Choose Mode 2 (Cloud-Native) if you are on AWS/GCS/Azure. It is much easier to manage, cheaper (no need for 3x local replication), and the cache is very efficient. Set your cache size to fit about 7 days of data, and StarRocks handles the rest. * Choose Mode 1 (Shared-Nothing) only if you are On-Premises or have extremely strict latency requirements that cannot tolerate even occasional S3 "cold" reads. Production Tip: If you choose Mode 2, ensure your CN nodes have high-performance Local NVMe SSDs for the cache. This ensures that the "hot" 7 days of data perform exactly like local storage. References • introduction/Architecture.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/s3.md • table_design/table_design.md • administration/management/FE_configuration.md References • introduction/Architecture.md • deployment/shared_data/feature-support-shared-data.md

@Rocky Mode 2 (Cloud-Native) - is the data first written to s3 first and then to the cache, or does it write to s3 when cache ttl expires?

In Mode 2 (Cloud-Native Shared-Data), the write sequence depends on your configuration, but it never waits for a TTL to expire before writing to S3. S3 is the primary "source of truth" and data must reach it to be considered persistent. Here is how the two write paths work: 1. Simultaneous Write (Default) When

datacache.enable

is set to

true

(the default), the data is written simultaneously to both S3 and the local SSD cache. * The Flow: Compute Node (CN) receives data → Writes to Local SSD AND starts uploading to S3 → Waits for S3 to confirm receipt → Returns "Success" to the user. * Result: Data is immediately durable in S3 and immediately "hot" in your local cache for the next query. 2. Asynchronous Write (High Performance) StarRocks supports an "Async Write-back" mode (often enabled via

enable_shared_storage_async_write

in newer versions or seen as "Asynchronous data ingestion" in docs) to hide S3 latency. * The Flow: CN receives data → Writes to Local SSD cache → Immediately returns "Success" to the user → Background thread uploads the data to S3. * Result: Much faster ingestion because the user doesn't wait for S3 network latency. * Risk: If a CN node physically dies before the background upload completes, the data from that specific un-uploaded batch could be lost (similar to how local disk buffers work). What about the 7-day TTL? The

datacache.partition_duration

(e.g.,

7 DAY

) does not control when data moves to S3. It controls the cache eviction priority: * Initial Write: All data is written to S3 immediately (or shortly after with async write). * Cache Logic: The data stays on your local SSD as a cache. If the SSD gets full, StarRocks uses an LRU (Least Recently Used) policy to clear space. * The TTL Rule: If you set a 7-day duration, StarRocks will keep data younger than 7 days "hot." If that data is evicted due to space, and you query it again, StarRocks will automatically pull it back from S3. If data is older than 7 days and gets evicted, StarRocks will generally not prioritize re-caching it unless configured otherwise. Summary * Is it written to S3 first? In default mode, it's simultaneous. In async mode, it's local first, then S3 (usually within seconds). * Does it wait for TTL to expire? No. Data is sent to S3 as part of the ingestion process. The TTL only manages how long that data "clings" to your local SSD for speed. References • deployment/shared_data/s3.md • administration/management/BE_configuration.md • introduction/Architecture.md • deployment/shared_data/azure.md • using_starrocks/caching/block_cache.md References • deployment/shared_data/s3.md • knowledge/usage/load/shared_data_batch_ingestion.md

@Rocky 1. If I am not mistaken querying iceberg tables on s3 via exrternal catalog is supported only in Classic Shared-Nothing mode? 2. Why online we see that most of the companies use this mode (Classic Shared-Nothing with external catalog) Does this mean that Classic Shared-Nothing without iceberg is not recommended?

1. Support for Iceberg in Cloud-Native Mode Actually, querying Iceberg tables (or any external catalog like Hive/Hudi/Delta) is fully supported in both modes: * Classic Shared-Nothing: The Backends (BEs) handle both the internal data and the external S3 data. * Cloud-Native Shared-Data: The Compute Nodes (CNs) handle both the internal "cloud-native" tables and the external Iceberg tables. The Data Cache mechanism in Mode 2 (Cloud-Native) is actually specifically designed to accelerate these external queries. When you query an Iceberg table on S3, the CN nodes cache the remote Parquet/ORC files locally on their SSDs just like they do for internal tables. 2. Why do most companies use Classic Shared-Nothing with External Catalogs? You see this pattern frequently online for three main reasons: * Maturity & History: Shared-Nothing was the only architecture available in StarRocks for years (pre-v3.0). Most established production deployments and case studies you see today were built before the Cloud-Native mode was released. * On-Premises Infrastructure: Many large enterprises run StarRocks on-premises (IDCs) where "S3-style" shared storage is either unavailable or slower than their local NVMe drives. In these cases, Shared-Nothing is the natural choice. * The "Lakehouse" Entry Point: Many users start by using StarRocks simply as a fast query engine on top of their existing Iceberg/Hive lake. For this "federated" use case, they often just spin up a standard cluster (Shared-Nothing) because it's the most common starting point in the docs. 3. Is Shared-Nothing without Iceberg "Not Recommended"? No, quite the opposite. Using StarRocks with its Internal Native Tables (the StarRocks segment format) in Shared-Nothing mode is the "Gold Standard" for performance. While Iceberg is great for open standards, StarRocks' Native Tables are almost always faster because: 1. CBO Optimization: The Cost-Based Optimizer has deeper statistics for internal tables. 2. Indexing: You get advanced indexing (Bitmap, Bloom Filter, Inverted Indexes) that doesn't exist in the same way for Iceberg. 3. Compaction Control: StarRocks background processes continuously optimize internal data for read speed. Summary Recommendation * Use Native Tables (Shared-Nothing or Shared-Data): When you need sub-second response times for dashboards and high-concurrency apps. * Use Iceberg Catalogs: When you need to join your high-performance StarRocks data with your massive historical data lake, or when data is being written by other engines (like Spark/Flink). Verdict: Both architectures are stable. The trend is moving toward Mode 2 (Cloud-Native) for anyone on AWS/GCP/Azure because it simplifies scaling, while Mode 1 (Shared-Nothing) remains the powerhouse for On-Prem and maximum-performance "Hot" storage. References • deployment/shared_data/feature-support-shared-data.md • data_source/catalog/iceberg/iceberg_catalog.md • introduction/Architecture.md • data_source/catalog/catalog_overview.md References • deployment/shared_data/feature-support-shared-data.md • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md

@Rocky Shared-Nothing is more production ready than Cloud-Native? is there any known bugs on any of them?

Both architectures are production-ready, but they have different levels of maturity and different focus areas. 1. Maturity vs. Production Readiness * Shared-Nothing (Classic): This is the more mature architecture. It has been the core of StarRocks since its inception (v1.x). It is "battle-tested" across thousands of diverse production environments. If your definition of "production ready" is "longest track record with the most edge cases discovered and fixed," then Shared-Nothing is the winner. * Cloud-Native (Shared-Data): This is stable and production-ready for core OLAP workloads as of v3.1+. It is the default for StarRocks' own managed cloud service. While newer, it has been hardened significantly. However, it does not yet have 100% feature parity with Shared-Nothing. 2. Known Limitations & Gaps (As of v3.3/v3.4) While "bugs" are constantly being fixed in every release, there are known architectural gaps in Cloud-Native mode that you should consider: Feature | Shared-Nothing | Cloud-Native (Shared-Data) Backup & Restore | Fully supported (

BACKUP

/

RESTORE

SQL) | Not supported (You must use S3-level replication/snapshots) Inverted Index | Fully supported (v3.0+) | Limited support (Being added in recent v3.3.x+ versions) Data Rebalance | Required (moves physical data) | Not required (CNs are stateless; scaling is instant) Table Types | All supported | Primary Key, Duplicate, Aggregate (All major types supported) Storage Usage | Precise | Can appear inflated on S3 due to historical versions/lagging vacuum 3. Common "Friction Points" (Not necessarily bugs, but operational challenges) In Cloud-Native Mode: * Compaction Staleness: Since compaction is coordinated by the FE and executed by CNs, tasks can sometimes get "stuck" if a CN node is restarted frequently or if resources are tight. This can lead to a build-up of small files on S3. * Cache Warming: After a cluster restart or scaling, query performance might be slow for the first few minutes until the "hot" data is pulled from S3 back into the local SSD cache. * S3 Latency: If your S3-compatible storage (like MinIO) is unstable or under high load, it will directly impact ingestion and query performance for "cold" data. In Shared-Nothing Mode: * Tablet Balancing: When adding a new node, StarRocks must physically move tablets (data files) from old nodes to new ones. In very large clusters (100TB+), this process requires careful monitoring of disk I/O and network. * Disk Failures: You are responsible for monitoring local SSD health. If a disk fails, StarRocks must rebuild the missing replicas from other nodes. Summary Verdict * Choose Shared-Nothing if you require the most "vetted" code path, need built-in

BACKUP/RESTORE

commands, or are running on hardware where you manage your own disks. * Choose Cloud-Native if you want the operational simplicity of S3 durability (no more worrying about disk-level replication) and the ability to scale your compute nodes up and down in seconds without moving data. Recommendation: For most modern cloud deployments, the benefits of Cloud-Native (elasticity and S3 durability) now outweigh its relative youth compared to Shared-Nothing. If you choose Cloud-Native, just ensure you are on v3.2 or later for the most stable experience. References • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • introduction/Architecture.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md References

@Rocky can we have a mix mode ? meaning it has starrocks tables in s3 but also on starrocks disk ?

No, a mixed mode is not supported within a single cluster. When you set up StarRocks, you must choose one architecture via the

run_mode

configuration (

shared_nothing

or

shared_data

). However, you can achieve a "mixed feel" in both modes through different features: 1. In Cloud-Native (Shared-Data) Mode While the source of truth for all tables is S3, you effectively have the data on disk too. * The "Disk" part: Every node (CN) has a local SSD cache. When you query or write data, StarRocks stores it on the local disk. * Performance: If your "hot" data fits in your local SSDs, StarRocks will read it from the disk exactly like the Shared-Nothing mode. It only goes to S3 for "cold" data that has been evicted from the cache. * Best for: When you want the convenience of S3 (easy scaling, high durability) but the speed of local disks for your active data. 2. In Classic (Shared-Nothing) Mode The source of truth for your internal tables is the local disk, but you can still use S3. * Internal Tables: Stored strictly on the BE nodes' local disks (replicated for safety). * External Tables (S3/Iceberg): You can create an External Catalog to query data residing on S3 (Iceberg, Hive, Delta Lake, or even raw Parquet). * Best for: When you have a core set of "mission-critical" high-speed tables on local disks and want to join them with massive, cheaper datasets sitting in a data lake on S3. Summary Comparison Feature | Shared-Nothing | Cloud-Native (Shared-Data) Cluster Mode |

run_mode = shared_nothing

|

run_mode = shared_data

Native Tables | Stored on Local BE Disks | Stored on S3 + Cached on CN Disks External S3 Tables | Supported (via Catalog) | Supported (via Catalog) Can you mix? | No. One cluster, one mode. | No. One cluster, one mode. Important Note: You cannot change a cluster from one mode to the other after it is created. You would need to set up a new cluster and migrate the data. References • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md References • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md References • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • administration/management/FE_configuration.md