http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am looking into shared-data and D...
# questions-and-troubleshooting
a
@Rocky I am looking into shared-data and Data Cache. What is the recommended starting point on: • tuning 
datacache_disk_size
and 
starlet_star_cache_disk_size_percent
• additional overhead we should reserve for filesystem overhead and other local-disk usage? • how does numbers of CN / expected data scale impact this?
r
For tuning Data Cache in StarRocks shared-data (compute-storage separation) clusters, the following guidelines apply to version 3.x and later. 1. Tuning 
datacache_disk_size
vs 
starlet_star_cache_disk_size_percent
 In a shared-data cluster, StarRocks calculates the total cache capacity by taking the greater of these two values. * 
starlet_star_cache_disk_size_percent
(Default: 80): This is the most common way to configure cache. It sets the limit as a percentage of the disk's total capacity. * 
datacache_disk_size
(Default: 0): Used to set a fixed physical limit (e.g., 
500G
, 
2T
). If this is set to 
0
, the system relies entirely on the percentage parameter. * Recommendation: Use the default 80% as your starting point. This ensures that the cache can grow with your disk size if you resize volumes later. Only use 
datacache_disk_size
if you have very large disks and want to explicitly cap the cache size to avoid long metadata loading times or to reserve specific amounts of space for other processes. 2. Recommended Overhead Reservation The remaining 20% (when using the 80% default) is critical for several background and runtime processes: * Compaction Peaks: During compaction, StarRocks may temporarily double the disk space usage for affected data segments (e.g., 100GB of data can peak at 200GB during the merge process before the old version is garbage-collected). * Persistent Indexes: If you use Primary Key tables, the persistent index files are stored locally in 
${storage_root_path}/persist/
. These are not part of the Data Cache but occupy the same disk. * Spilling: When complex queries (Joins, Aggregations) exceed memory limits and 
enable_spill
is true, intermediate data is spilled to local disk. * Logs and Core Files: Regular system logs (
<http://be.INFO|be.INFO>
) and potential core dumps in case of a crash need a safety buffer to prevent the disk from reaching 100%, which would cause the node to stop functioning. * Filesystem Metadata: Filesystems like XFS or EXT4 require a small percentage of free space to maintain performance and avoid fragmentation. 3. Impact of CN Count and Data Scale * Total Cluster Cache: Because Data Cache is node-local, your total cluster cache capacity is 
(Cache per CN) * (Number of CNs)
. * Data Scale (Hot vs. Cold): The ideal scale is to have your "hot" working set (e.g., the last 7 days of data) fit entirely within the combined cache of all CNs. If your cache hit rate is consistently < 50%, it is a signal to either add more CNs or increase the disk size per CN. * Elastic Scaling: When you add/remove CNs, data routing changes. This can cause "cache misses" as the new nodes' caches are empty. * Tip: To mitigate this, enable Cache Sharing (set 
enable_datacache_sharing = true
and FE config 
enable_trace_historical_node = true
). This allows CNs to fetch cached data from one another via the network before falling back to remote object storage. * Disk Usage Consistency: You may notice uneven disk usage across CN nodes; this is normal behavior as cache is populated on-demand based on the specific queries each node handles. Monitoring & Maintenance * Peak Usage: Note that the Data Cache directory may appear larger (via 
du
) than the actual cached data because it reflects historical peak usage. Eviction marks blocks as "overwritable" rather than immediately deleting files. * Clearing Cache: If you need to clear the cache for emergency maintenance, you can manually delete the 
datacache/
sub-directory under your 
storage_root_path
and restart the CN. References • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md
a
@Rocky 1. What is difference of 
starlet_star_cache_disk_size_percent
 and 
datacache_disk_size
 ? 2. What are the size we should consider "super large" to use 
datacache_disk_size
 , if we still look for percent based configuration? 3. Do we need to reserve additional overhead on top of 
starlet_star_cache_disk_size_percent
 ?
r
1. Difference and Interaction In a shared-data (compute-storage separation) cluster, StarRocks uses both parameters to determine the cache capacity on each node. * 
starlet_star_cache_disk_size_percent
: Sets the limit as a percentage of the disk's total capacity (Default: 80). * 
datacache_disk_size
: Sets the limit as a fixed value (e.g., 
500G
, 
2T
) or a specific percentage (e.g., 
80%
). * Priority Rule: The system calculates the cache capacity based on both and then takes the greater value. For example, if you have a 1TB disk, the default 80% (
starlet...percent
) is 800GB. If you manually set 
datacache_disk_size
to 900GB, the system will use 900GB. 2. When to use 
datacache_disk_size
over Percentage There isn't a single "super large" threshold (e.g., 10TB) defined in the docs, but you should switch from percentage-based to fixed-size (
datacache_disk_size
) in the following scenarios: * Heterogeneous Disks: If your CN nodes have different disk sizes but you want a uniform cache size across the fleet for predictable performance. * Preventing "Warm-up" Delays: With extremely large disks (e.g., 20TB+), an 80% cache results in 16TB of metadata. On restart, loading the metadata for 16TB of cached blocks can take significant time. If your "hot" data is only 2TB, it is better to cap the cache at 2TB or 3TB using 
datacache_disk_size
rather than letting it fill the whole disk. * Shared Environments: If the local disk is shared with other applications or heavy system logs, a fixed cap prevents StarRocks from starving those processes. 3. Overhead Requirements You do not typically need to reserve more than the default 20% (the gap left by the 80% default), but you must verify that the following "non-cache" items fit within that 20%: * Primary Key Indexes: If using PK tables, the persistent index files are stored in 
${storage_root_path}/persist/
. These can grow large depending on your unique key count and are not part of the Data Cache. * Query Spilling: If queries exceed memory and spill to disk (
enable_spill = true
), they use the remaining disk space. * Compaction: Compaction requires temporary workspace. During a merge, a segment can effectively double in size on disk before the old version is deleted. * Logs and Cores: System logs and core dumps (in case of crashes) need a safety buffer. Recommendation: Start with the default 80%. If you notice your disks hitting 90-95% (which can cause filesystem performance degradation or node hangs), reduce the percentage to 70% or use a fixed 
datacache_disk_size
. 4. Impact of CN Count and Data Scale * CN Count: Total cluster cache = 
(Cache per CN) * (Number of CNs)
. Adding CNs linearly increases your total "hot data" capacity. * Scale: If your total "hot" dataset (e.g., last 30 days of data) is 10TB and your total cluster cache is only 5TB, you will experience a high "cache miss" rate and degraded query performance. * Consistency: Disk usage will likely be uneven across CNs because cache is populated based on which tablets each node is processing. This is expected behavior and not an indicator of imbalance unless one node is at 100% and others are at 10%. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache.md
a
@Rocky when designing numbers of nodes for CN, what resource usage type should be the most demanding for CN in shared-data cluster?
r
When designing the number of Compute Nodes (CN) for a StarRocks shared-data cluster, CPU processing power is typically the most demanding and critical resource for sizing. While all four main resources (CPU, Memory, Disk, and Network) are important, they play different roles in a shared-data architecture: 1. CPU: The Primary Bottleneck StarRocks is a high-performance MPP (Massively Parallel Processing) database. In a shared-data cluster, the CN node is stripped of persistent storage responsibilities, leaving it to focus almost entirely on computation. * Query Complexity: CPU is the primary driver for handling vectorized execution, complex joins, aggregations, and data decompression. * Background Tasks: CNs are responsible for compaction, which is a CPU-intensive process of merging small files in remote storage into larger ones. If CPU is undersized, compaction will lag, leading to a "small file trap" that degrades query performance. * Estimation Rule: Performance for analytical queries scales linearly with CPU. The total number of nodes is often calculated based on the estimated number of rows to be processed per second per core. 2. Memory: The Execution Workspace Memory is the second most critical resource and should be scaled in a fixed ratio to your CPU. * Recommended Ratio: The standard recommendation is a 1:4 ratio (e.g., 16 vCPUs to 64GB RAM). * Demand Drivers: Memory is heavily consumed by intermediate result sets during large joins and by the Primary Key persistent index (if you use PK tables). While StarRocks can spill to disk if memory is exhausted, doing so significantly slows down performance. 3. Local Disk (Data Cache): The Performance Accelerator While the CN is "stateless," it uses local disks as a Data Cache to turn remote object storage (S3/HDFS) into "hot" local storage. * IOPS & Throughput: High-speed NVMe or SSDs are highly recommended. While disk capacity determines how much "hot" data you can hold, the IOPS/Throughput of the disk is what prevents the CPU from waiting on I/O during cache hits. 4. Network: The Data Pipe * In-Cluster Traffic: High bandwidth (10Gbps minimum, 25Gbps+ recommended) is necessary for shuffling data between nodes during distributed joins and for pulling data from object storage. * Elasticity: In shared-data mode, data is pulled from the network rather than read from local persistent disks. This makes network bandwidth a more frequent bottleneck here than in traditional shared-nothing (BE) setups. Summary Checklist for Sizing CNs: Resource | Priority | Sizing Strategy CPU | Highest | Size based on query concurrency and total rows processed per second. Memory | High | Keep a **1:4 core-to-RAM ratio**; increase if using massive PK tables. Disk | Medium | Use **NVMe/SSD**; capacity should cover your "hot" working set (e.g., last 30 days of data). Network | Medium | Ensure 25Gbps if you have high-volume ingestion or frequent remote storage scans. Recommendation: Start with nodes having at least 16 cores and 64GB RAM. If your query profile shows high 
IOTimeRemote
and low CPU usage, you likely need more/better local disk cache. If CPU usage is consistently >80%, it is time to scale out the number of CN nodes. References • deployment/plan_cluster.md • deployment/deploy_manually.md • knowledge/usage/best-practice.md • quick_start/shared-data.md • introduction/Architecture.md
a
@Rocky What about FE node?
r
For a StarRocks FE (Frontend) node, especially in a shared-data cluster, the most demanding resource type is Memory (RAM). While CN nodes are CPU-bound because they handle the heavy lifting of data processing, FE nodes are memory-bound because they must store and maintain the entire cluster's metadata in-memory. 1. Memory: The Most Critical Resource Every FE node (Leader, Follower, and Observer) maintains a complete copy of the cluster's metadata in its JVM heap. As your data grows—specifically the number of tables, partitions, and tablets—your FE memory consumption will increase linearly. Sizing Guide for FE Memory: The required RAM is directly tied to the number of tablets in your system: Total Tablets | Recommended FE RAM < 1 Million | 16 GB 1 – 2 Million | 32 GB 2 – 5 Million | 64 GB 5 – 10 Million | 128 GB Note: You must configure 
-Xms
and 
-Xmx
in 
fe.conf
to match these values. 2. Shared-Data Specific Overhead In a shared-data cluster, the FE has additional responsibilities that add slight overhead to its resource profile: * Lake Compaction Scheduling: The FE is responsible for monitoring "compaction scores" of data in object storage and scheduling compaction tasks for the CN nodes. * Metadata Synchronization: The FE regularly synchronizes metadata with the remote object store to ensure consistency across the compute layer. * Garbage Collection: The FE manages the deletion of obsolete data files in S3/HDFS. If you have an "excessive number of shards to be cleaned," this can put temporary pressure on the FE JVM. 3. CPU: For Query Planning and Concurrency * Planning: FE uses CPU for parsing SQL, optimizing query plans, and managing client connections. * Baseline: 8 Cores is the standard starting point. * Scaling: If you have thousands of concurrent users or extremely complex SQL (long planning times), you may need more cores or additional Observer FE nodes to offload the planning work from the Leader FE. 4. Disk: For Metadata Persistence The FE uses a local disk to store its image and edit logs (via BDB JE). * Size: 100 GB is typically sufficient for metadata logs. * Type: SSD/NVMe is highly recommended. While the FE doesn't store data files, the speed at which it can flush metadata logs and recover during a restart depends on disk I/O performance. Summary Recommendation for FE * Priority: Memory > CPU > Disk. * Standard Spec: 8-16 Cores, 32-64 GB RAM, 100 GB SSD. * HA Note: Deploy exactly 3 Follower FEs for high availability. Adding more nodes for performance should be done via Observer FEs. References • deployment/plan_cluster.md • faq/Others.md • administration/management/FE_configuration.md • introduction/Architecture.md
a
@Rocky what Tablets is referring to here?
r
In StarRocks, a Tablet is the smallest unit of data management and distribution. When you hear about "Tablets" in the context of FE (Frontend) memory sizing, it refers to the total number of these data shards across your entire cluster. Here is the breakdown of what they are and why they impact the FE: 1. The Hierarchy You can think of data organization like a tree: * Table: The top-level object. * Partition: A table is split into partitions (e.g., by 
event_date
). * Tablet: Each partition is further divided into Buckets. One bucket of data in one partition is a Tablet.
Formula: 
Number of Partitions
× 
Number of Buckets
= 
Total Tablets
2. Why the FE Cares About Tablets The FE acts as the "brain" of the cluster. It doesn't store the actual data (that's for CN/BE), but it must keep the metadata for every single tablet in its RAM. For each tablet, the FE tracks: * Location: Which nodes are currently caching or processing this tablet. * Version: The current data version (to ensure queries see the latest committed data). * Health: Whether the tablet is healthy or needs repair/compaction. As the number of tablets grows into the millions, this "catalog" of metadata consumes significant JVM heap memory. This is why the FE RAM requirements scale directly with the tablet count. 3. Shared-Data vs. Shared-Nothing * In Shared-Nothing (Classic BE): Each tablet has multiple replicas (usually 3). The FE tracks the location of every replica. * In Shared-Data (CN Cluster): There is only one logical tablet (since data is safe in S3/HDFS), but the FE still tracks which CN nodes are currently managing that tablet's cache and metadata. 4. Best Practices for Tablet Management To keep your FE healthy and avoid OOM (Out of Memory) errors: * Target Size: Aim for each tablet to hold about 1 GB to 10 GB of data. * Avoid "Small Tablet" Problem: If you have 1,000 partitions and 100 buckets per partition, you have 100,000 tablets. If each tablet only has 1MB of data, you are wasting massive amounts of FE memory on metadata for very little data. * Monitoring: You can check your current tablet count by running:
Copy code
sql
    SHOW PROC '/statistic';
Or check the 
be_tablets
table in the 
information_schema
. References • faq/Others.md • administration/management/monitoring/metrics.md • knowledge/trouble_shooting/fe_memory_problems.md • table_design/table_design.md • administration/management/resource_management/Memory_management.md • using_starrocks/caching/query_cache.md • sql-reference/information_schema/be_tablets.md
a
@Rocky for shared-data , what resource / nodes will be more dependent on handle PK tables mutations and compaction?
r
In a shared-data architecture, handling Primary Key (PK) table mutations (upserts/deletes) and compaction shifts the burden almost entirely onto the Compute Nodes (CN). While the FE manages the schedule, the CN is where the actual resource-intensive work happens. Here is the breakdown of which resources are most dependent on these processes: 1. The Most Dependent Node: CN (Compute Node) The CN is the workhorse for PK tables. Unlike standard Duplicate or Aggregate tables, PK tables require a "read-on-write" or "index-lookup" operation during every mutation. * Resource: CPU (High Demand) * Compaction: Compaction is a background process that merges small files into larger ones. For PK tables, this is CPU-intensive because the system must calculate and generate "delete vectors" to track which rows have been superseded by newer versions. * Mutations: Every upsert/delete requires the CN to find the location of the old key using the PK index, which involves CPU cycles for hashing and index traversal. * Resource: Memory (High Demand) * PK Index: The PK index (mapping of 
Key
-> 
Row Location
) is stored on the CN. Even if you use a Persistent Index, a portion of it must be cached in the CN's memory (
l0_max_mem_usage
) to maintain ingestion performance. * Memtables: During mutations, data is first written into memory-resident memtables before being flushed to object storage. * Resource: Local Disk / SSD (Critical for Performance) * Persistent Index: In shared-data mode, you should use 
persistent_index_type = "CLOUD_NATIVE"
. This stores the full index on S3 but uses the local SSD of the CN as a high-speed cache. If your local disk is slow (HDD) or has low IOPS, ingestion and compaction performance will tank. 2. The Orchestrator: FE (Frontend) The FE's role in PK mutations and compaction is primarily metadata management and scheduling. * Resource: Memory (Moderate Demand) * The FE tracks "Compaction Scores" for every partition. If a partition has too many un-compacted versions, the FE identifies it and assigns a compaction task to an available CN. * Because PK tables often involve many small transactions (e.g., real-time streaming), the FE may experience higher Edit Log activity as it commits these transactions. * Resource: CPU (Low Demand) * The FE is not involved in the actual data merging or index lookups; it only handles the transaction logic (Commit/Publish). Summary Table: Resource Dependency Task | Primary Node | Most Demanding Resource | Why? PK Mutation | CN | Memory & CPU | Index lookups and maintaining delete vectors in RAM. Compaction | CN | CPU & Local SSD | Merging files and updating persistent index cache on local disk. Scheduling | FE | Memory | Tracking metadata versions and tablet/partition health. Sizing Advice for PK Tables in Shared-Data: 1. CN Memory: Ensure your CNs have enough RAM to hold the "hot" portion of your PK index. If the index doesn't fit in the allocated memory, StarRocks will swap to disk, slowing down ingestion. 2. CN Disk: Use high-performance NVMe SSDs for the local data cache. This is where the Persistent Index and the Data Cache live. 3. Compaction Threads: Adjust 
compact_threads
on the CN (usually 25-50% of CPU cores) to ensure compaction can keep up with your ingestion rate without starving your queries. References • table_design/table_types/primary_key_table.md • administration/management/compaction.md • deployment/shared_data/feature-support-shared-data.md • best_practices/primarykey_table.md
a
@Rocky assuming we are bucketing and partitioning a table with 10 TB size, what are the target size for bucket / partition we should be aimming to?
r
For a table of 10 TB (raw data), sizing your partitions and buckets correctly is critical to balancing query performance, background maintenance (compaction), and Frontend (FE) stability. Here are the target sizes and the logic behind them based on StarRocks best practices: 1. Target Tablet (Bucket) Size: ~10 GB Raw Data In modern StarRocks versions (v2.4+), the recommended target size for a single tablet is 10 GB of raw data. * Why 10 GB? StarRocks can scan a single tablet using multiple threads in parallel (
enable_tablet_internal_parallel = true
). This allows you to have fewer, larger tablets, which reduces the metadata overhead on the FE while still achieving high scan speeds. * Historical Note: You may see older docs suggesting 100 MB – 1 GB. Those usually refer to compressed data or older versions without internal parallel scanning. For a 10 TB scale, stick to the 10 GB raw rule. 2. Target Partition Size: ≤ 100 GB Raw Data While there is no hard limit, a good rule of thumb is to keep each partition at or below 100 GB. * Why 100 GB? Partitions are the unit of data management (TTL, recovery, and pruning). If a partition is too large, operations like 
DROP PARTITION
or re-balancing become very heavy. * Pruning: Partitions should be sized based on your query patterns (e.g., daily or monthly). If you query "last 7 days" of data, having daily partitions allows the engine to skip the other 9.9 TB entirely. 3. Sizing a 10 TB Table: Concrete Example Assuming your 10 TB table grows over time (e.g., 3 years of data), here is how you would calculate your distribution: Metric | Calculation | Target Value Total Partitions | 10,000 GB / 100 GB per partition | ~100 Partitions Total Tablets | 10,000 GB / 10 GB per tablet | ~1,000 Tablets Buckets per Partition | 1,000 Tablets / 100 Partitions | 10 Buckets Scenario A: 100 GB per Day (approx. 3 months of data) * Partition: Daily * Buckets: 10 (
DISTRIBUTED BY HASH(...) BUCKETS 10
) Scenario B: 10 GB per Day (approx. 3 years of data) * Partition: Daily * Buckets: 1 or 2 (to stay close to the 10 GB tablet target) 4. Critical Red Flags to Avoid * Too many tablets: Avoid exceeding 200,000 tablets per BE/CN node. For 10 TB, you are well within safe limits with 1,000–5,000 tablets. * Too many partitions: Avoid exceeding 100,000 total partitions in the cluster. This will cause the FE to run out of memory (OOM) because it tracks every partition in RAM. * Data Skew: Ensure your 
DISTRIBUTED BY HASH
column has high cardinality (like a 
user_id
or 
order_id
). If 90% of your data ends up in 1 bucket, that 10 GB tablet will become a 1 TB "whale" tablet, causing slow queries and compaction failures. Summary Checklist 1. Aim for ~10 GB raw data per tablet. 2. Aim for ≤ 100 GB raw data per partition. 3. Use 
PARTITION BY
 on a time column to enable data pruning. 4. Verify setting: Ensure 
SET GLOBAL enable_tablet_internal_parallel = true;
is enabled to get the best performance from 10 GB tablets. References • best_practices/partitioning.md • table_design/data_distribution/Data_distribution.md • best_practices/query_tuning/schema_tuning.md • administration/management/resource_management/Memory_management.md References • best_practices/partitioning.md • best_practices/query_tuning/schema_tuning.md • knowledge/admin_and_operation/shared_data_compaction.md
a
@Rocky can we scale up the numbers of distributed partition when data grow?
r
Yes, you can scale the number of partitions and buckets as your data grows. However, the method depends on whether you are adding new time-based partitions or trying to increase the bucket count (distribution) for data you already have. Here is how you handle scaling for both: 1. Scaling the Number of Partitions This is the standard way to handle growth. As time progresses, you simply add new partitions. * Manual: You can use 
ALTER TABLE ... ADD PARTITION
. * Dynamic Partitioning: You can configure StarRocks to automatically create new partitions (e.g., daily) and drop old ones. As your data volume grows, the cluster naturally handles more partitions. _ _Limit*: Avoid exceeding 100,000 total partitions across the cluster to keep FE memory healthy. 2. Scaling the Number of Buckets (Distribution) If your data volume per partition increases (e.g., you used to get 10GB/day but now get 100GB/day), you will need more buckets to keep tablet sizes around the 10GB target. A. For New Partitions (Recommended) Starting from v3.5.8 and v4.0.1, you can change the default bucket count for all future partitions without affecting existing data. This is the most efficient way to scale up as your ingestion rate grows.
Copy code
sql
-- This sets the bucket count to 20 for all partitions created from now on
ALTER TABLE your_table DISTRIBUTED BY HASH(user_id) DEFAULT BUCKETS 20;
B. For Existing Partitions (Shared-Nothing/BE only) In Shared-Nothing clusters (v3.2+), you can trigger a background "re-bucketing" of existing data. StarRocks will rewrite the data into the new bucket count.
Copy code
sql
-- Re-bucket specific partitions
ALTER TABLE your_table PARTITIONS (p20231201, p20231202) 
DISTRIBUTED BY HASH(user_id) BUCKETS 20;

-- Re-bucket the entire table (all partitions)
ALTER TABLE your_table DISTRIBUTED BY HASH(user_id) BUCKETS 20;
Note: This operation is resource-intensive as it involves reading and rewriting the data. C. For Shared-Data (CN Cluster) As of current versions, Shared-Data mode does not support re-bucketing existing data via 
ALTER TABLE
. To scale existing shared-data tables, you would typically use 
INSERT INTO new_table SELECT * FROM old_table
. However, you can still use the 
DEFAULT BUCKETS
method mentioned above for new data. 3. Automatic Scaling (Random Bucketing) If you use Random Bucketing (available for Duplicate Key tables) and set a 
bucket_size
, StarRocks can dynamically increase the number of buckets during data loading based on the volume of data being ingested.
Copy code
sql
CREATE TABLE logs ( ... )
DISTRIBUTED BY RANDOM
PROPERTIES("bucket_size" = "1073741824"); -- Aim for 1GB per bucket
This is the closest to "auto-scaling" distribution, as the system handles the bucket math for you as the load grows. Summary Recommendation * If data grows over time: Use Dynamic Partitioning to add more partitions. * If data volume per day grows: Use 
ALTER TABLE ... DEFAULT BUCKETS
(v3.5.8+) to increase the bucket count for future partitions. * If you have serious skew or "whale" partitions: Use 
ALTER TABLE ... BUCKETS
(v3.2+) to re-distribute those specific partitions (Shared-Nothing only). References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • table_design/data_distribution/Data_distribution.md • best_practices/bucketing.md • best_practices/partitioning.md • table_design/data_distribution/automatic_bucketing.md References
a
@Rocky in shared data, when copying from iceberg table to starrocks hot cache, what is the usual / expected change in size in the same data on disk?
r
In a shared-data architecture, the "hot cache" refers to the Data Cache (also known as Block Cache) on the local SSDs of the Compute Nodes (CN). The change in size depends on whether you are querying the Iceberg table directly or ingesting (copying) the data into a native StarRocks table. 1. Querying External Iceberg (using Data Cache) If you are simply querying an Iceberg table via an External Catalog, the Data Cache fetches and stores blocks of the original remote files (Parquet/ORC). * Size Change: 0% (Identical size). * Reason: The Data Cache is a "Block Cache." It splits the remote Parquet/ORC files into fixed-size blocks (e.g., 1 MB) and stores them on the local SSD exactly as they are in object storage. If your Iceberg Parquet files are compressed with ZSTD or Snappy, the Data Cache stores those compressed blocks. There is no re-compression or decompression at this layer. 2. Ingesting Iceberg into a StarRocks Table If you are "copying" data by running 
INSERT INTO sr_table SELECT * FROM iceberg_table
, the data is converted from Iceberg's format into StarRocks’ native Segment format. * Size Change: Typically 10% to 30% reduction. * Reason: StarRocks’ native storage format is highly optimized for OLAP. While both Parquet and StarRocks use columnar storage and algorithms like ZSTD/LZ4, StarRocks often achieves better density due to: * Specialized Encodings: More aggressive use of dictionary encoding, bit-shuffling, and RLE (Run-Length Encoding) tailored for its execution engine. * Compaction: As StarRocks compacts data, it removes overlaps and deletes, further reducing the physical footprint compared to a collection of many small Parquet files. * Hot Cache impact: Once ingested, the Data Cache will now store blocks of these optimized StarRocks segment files. Because the segment files are smaller than the original Parquet files, the same amount of data will occupy less space in your "hot cache." Summary Table Action | Data Format in Cache | Expected Size vs. Iceberg Disk Size Direct Query (Catalog) | Original Parquet/ORC Blocks | 100% (No change) Ingestion (Internal Table) | StarRocks Segment Blocks | 70% – 90% (Usually smaller) Pro-Tip: Memory vs. Disk Cache It is important to distinguish the Data Cache (Disk) from the Page Cache (Memory): * Data Cache (Disk): Stores compressed blocks. * Page Cache (Memory): Stores decompressed data pages. If you have 1 TB of compressed data in your "hot cache" on disk, it may expand to 3–5 TB in memory (Page Cache) once queried and decompressed for processing. References • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md • table_design/data_compression.md • using_starrocks/caching/block_cache.md References • data_source/catalog/iceberg/iceberg_catalog.md • quick_start/iceberg.md • deployment/plan_cluster.md • best_practices/partitioning.md • data_source/External_table.md • knowledge/admin_and_operation/shared_data_compaction.md
a
@Rocky how starrocks decides which transaction it will compute in the queue first?
r
StarRocks handles "queues" differently depending on whether you are talking about general queries (SELECT/DML) or background ingestion jobs (like Broker Load). 1. General Queries and DML (Query Queues) For standard SQL transactions and queries, StarRocks uses a Query Queue mechanism (enabled via 
enable_query_queue_select
or 
enable_query_queue_load
). * Order of Execution: By default, queries are handled in First-In, First-Out (FIFO) order. When the cluster reaches a resource threshold (CPU, Memory, or Concurrency), new queries are placed in a 
PENDING
state. * The "First" Decision: As soon as resources (slots) become available on the Backend (BE), the query that has been waiting the longest in the queue is moved to the 
RUNNING
state. * Resource Groups: If you use Resource Groups, the queue is split. A query in a high-priority resource group won't necessarily "jump" the line of a global queue, but it will have its own dedicated "concurrency pool." This ensures that even if a "default" queue is full, queries in a specific resource group can still compute immediately if that group has available capacity. 2. Ingestion Jobs (Broker Load) For 
BROKER LOAD
jobs, StarRocks introduced a specific Priority system in v2.5. * Explicit Priority: You can specify a priority when creating the load:
Copy code
sql
    LOAD LABEL my_label (...)
    PROPERTIES ("priority" = "HIGH");
* The Decision: The Frontend (FE) scheduler looks at the 
priority
property. If multiple jobs are in the 
QUEUEING
state, those with higher priority are scheduled to start loading before those with lower priority, regardless of their submission time. 3. Transaction Commit Order For the final "commit" of a transaction (making data visible): * StarRocks processes commits sequentially at the FE level. * The order is determined by the completion time of the write tasks. Even if Transaction A started before Transaction B, if B finishes its data writing and sends the 
COMMIT
request first, it will be processed first. * There is no "priority" for the commit phase; it is strictly driven by which transaction reaches the "Ready to Commit" state first. Summary Table Job Type | Primary Decision Factor SELECT / INSERT | FIFO (First-In, First-Out) based on arrival time. BROKER LOAD | Priority Property (High/Medium/Low) > Submission Time. Resource Groups | Group Resource Limits (Isolation prevents one queue from blocking another). Commit Phase | Succession (Whichever finishes writing data first). How to Monitor You can see which transactions or queries are currently "waiting" in the queue by running:
Copy code
sql
SHOW RUNNING QUERIES; -- Look for the 'State' column (PENDING vs RUNNING)
References • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • faq/resource_isolation_faq.md • loading/SQL_transaction.md • administration/management/resource_management/resource_group.md References • faq/resource_isolation_faq.md • administration/management/resource_management/query_queues.md
a
@Rocky what are the suggestion to clean up staled transaction? we have ~1000 staled transaction (stream load & compaction) for last 3-4 days, they have reached the 1k lmax_running_txn_num_per_dbimit very easily, other than scale up max_running_txn_num_per_db i wnat to clean them up.
r
To clean up stale transactions that have reached the 
max_running_txn_num_per_db
limit (default 1000), you should first identify which transactions are stuck and then use the appropriate method to terminate them. 1. Identify Stale Transactions First, find the database ID and the list of running transactions: * Get DB ID: 
SHOW PROC '/transactions';
* List Transactions for a DB: 
SHOW PROC '/transactions/<db_id>';
(This shows states like 
running
, 
finished
, etc.) * View Specific Running Transactions: Use 
SHOW PROC '/transactions/<db_id>/running';
to see the IDs, labels, and start times. * Check Transaction Details:
Copy code
sql
    SHOW TRANSACTION WHERE ID = <transaction_id>;
Look at the 
PrepareTime
and 
TransactionStatus
. If it has been in 
PREPARED
or 
PREPARE
state for days, it is stale. 2. Clean Up Stream Load (2PC) Transactions If you are using the Stream Load Transaction Interface (common with Flink or Kafka integrations using Two-Phase Commit), transactions in the 
PREPARED
state must be manually committed or rolled back if the client fails to do so. To roll back a stale transaction via API:
Copy code
bash
curl -u <user>:<password> -X POST \
"http://<FE_IP>:<FE_HTTP_PORT>/api/transaction/rollback?db=<db_name>&label=<label>"
Note: If you have ~1000, you may need to script this using the labels found in the 
SHOW PROC
output. 3. Handle Stale Compaction Transactions Compaction transactions are usually managed internally. If they are stale for 3-4 days, it typically indicates a stuck process or a resource deadlock. * Check Compaction Status: 
SHOW PROC '/compactions';
* Shared-Data (Cloud-Native): If you are on a shared-data cluster, check 
select * from information_schema.be_cloud_native_compactions where STATUS != 'OK';
* Cleanup: Compaction transactions are generally tied to the FE's lifecycle. If they are truly stuck and blocking the transaction queue, restarting the Leader FE is often the most effective way to clear the internal transaction manager's memory of these stuck system-level transactions. 4. Adjust Timeouts to Prevent Recurrence To prevent transactions from staying open for 3-4 days in the future, check these FE configurations: * 
prepared_transaction_default_timeout_second
: (Default: 86400s / 1 day). This controls how long a 2PC transaction can stay in the 
PREPARED
state. You can reduce this to a few hours (e.g., 
14400
) so StarRocks automatically aborts them.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("prepared_transaction_default_timeout_second" = "14400");
* 
stream_load_default_timeout_second
: (Default: 600s). Controls the duration from 
BEGIN
to 
PREPARE
. 5. Managing the Limit If your workload naturally requires high concurrency, you can increase the limit temporarily to allow "good" transactions to pass while you clean the "bad" ones:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("max_running_txn_num_per_db" = "2000");
Summary Recommendation 1. Script a rollback for all Stream Load labels older than 24 hours using the HTTP API. 2. Restart the Leader FE if the stale transactions are primarily internal "compaction" types that do not clear after a few hours. 3. Lower the 
prepared_transaction_default_timeout_second
 to ensure the system self-cleans in the future. References • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • loading/Stream_Load_transaction_interface.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • faq/loading/Loading_faq.md
a
@Rocky what is the downside of setting s3_num_partitioned_prefix to maximum 1024 rather than default 256?
r
The 
aws.s3.num_partitioned_prefix
setting (introduced in v3.2.4) is used in StarRocks shared-data clusters to circumvent S3's request rate limits (3,500 PUTs and 5,500 GETs per second per prefix) by distributing data across multiple sub-paths. Setting this to the maximum of 1024 instead of the default 256 has no direct performance "penalty" within StarRocks itself, but there are several practical and operational downsides to consider: 1. Immutability (The Biggest Risk) Both 
aws.s3.enable_partitioned_prefix
and 
aws.s3.num_partitioned_prefix
are immutable once the Storage Volume is created. * If you set it to 1024 and later find that your S3 management tools or lifecycle policies struggle with that many directories, you cannot change it without creating a new Storage Volume and migrating your data. 2. S3 Management and Visibility "Noise" Increasing the prefix count creates a much more fragmented directory structure in your S3 bucket. * Default (256): Data is stored in paths like 
<s3://bucket/00/>
through 
<s3://bucket/FF/>
(2-digit hex). * Max (1024): Data is stored in paths like 
<s3://bucket/000/>
through 
<s3://bucket/3FF/>
(3-digit hex). * Impact: If you use the S3 Console or CLI (
aws s3 ls
) to inspect your data, you will have to navigate 1,024 folders. This makes manual troubleshooting, auditing, or "pointing-and-clicking" significantly more tedious. 3. S3 Lifecycle Policy Complexity If you use S3 Lifecycle rules to manage data (e.g., transitioning older StarRocks data to Cold Storage or Glacier), you might occasionally target specific prefixes. * Managing rules for 1,024 prefixes is more complex than 256. * While most users apply lifecycle rules at the bucket level, if your organization has "per-prefix" quotas or reporting, the administrative overhead scales with the prefix count. 4. Thinly Spread Data If your data volume is relatively small (e.g., only a few terabytes), spreading it across 1,024 prefixes means each prefix will hold very few files. * Listing Overhead: When StarRocks or external tools perform a recursive 
LIST
operation, S3's internal indexing of 1,024 distinct prefix nodes is technically more "work" than 256, though this is usually negligible for the StarRocks FE. Recommendation * Stick to 256 unless you are hitting 503 Slowdown errors from S3 during high-concurrency ingestion or heavy query peaks. * 256 prefixes provide a theoretical throughput of ~1.4 million GETs and ~896k PUTs per second, which is more than enough for the vast majority of StarRocks clusters. * Use 1024 only for extremely large-scale deployments (multiple petabytes) with thousands of concurrent users or continuous high-frequency Stream Loads. References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • deployment/shared_data/s3.md • faq/shared_data_faq.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • best_practices/partitioning.md • administration/management/FE_configuration.md
a
@Rocky does 512 a better option? let's say we have 400 datasets, 500K PUTs per seconds, what are the options?
r
For a workload requiring 500,000 PUTs per second, the choice between 512 and 1024 prefixes depends on your tolerance for S3 "Slow Down" errors (503s) and how balanced your ingestion is. 1. The Math: Prefixes vs. Throughput S3 supports a minimum of 3,500 PUT requests per second per prefix. Prefix Count | Theoretical Max PUTs/sec | Buffer for 500k PUT/sec Workload 256 (Default) | ~896,000 | ~1.8x (Thin buffer) 512 | ~1,792,000 | ~3.5x (Safe middle ground) 1024 | ~3,584,000 | ~7x (Maximum overhead) 2. Is 512 a Better Option? Yes, 512 is likely the "Sweet Spot" for your scale. * Safety against Skew: In a real-world cluster with 400 datasets, data is never perfectly distributed. If one large table is being heavily loaded, it might "hotspot" certain prefixes. A 3.5x buffer (512 prefixes) allows for significant skew without hitting S3 limits. * Reduced "Noise": 1,024 prefixes can be cumbersome to navigate in the S3 console or via CLI. 512 keeps the directory structure slightly more manageable (000 to 1FF in hex). * Power of 2: While the parameter accepts an integer, StarRocks internally uses a hash-and-modulo approach. Using a power of 2 (256, 512, 1024) is standard practice for uniform distribution in sharding. 3. Considerations for 500k PUTs/sec 500,000 PUTs per second is an extremely high volume for a database (typically implying millions of tiny files or massive parallelism). If you are actually hitting this rate, consider these options alongside the prefix count: * Batching (BE Config): Check your 
lake_publish_max_version_num_per_batch
. If you have many small transactions, StarRocks might be doing too many metadata PUTs. Increasing batch sizes reduces the total PUT count. * Segment Size: If you are generating 500k PUTs/sec due to many small segment files, increase your 
max_segment_file_size
(default 256MB) to ensure fewer, larger files are written to S3. * Consolidate Storage Volumes: Since you have 400 datasets, ensure they aren't all hitting the same S3 bucket prefix if you use multiple Storage Volumes. However, within a single Storage Volume, 
num_partitioned_prefix
is your primary tool. Summary Recommendation * Choose 512 if you want a robust safety margin for 500k PUTs/sec while keeping S3 management relatively "clean." * Choose 1024 only if your ingestion is extremely bursty (e.g., peaks significantly higher than 500k) or if you expect your data volume/concurrency to double again in the next year, as this setting is immutable once created. Note: Since this setting cannot be changed after the Storage Volume is created, if you have any doubt, overshooting to 1024 is safer than undershooting, as the only "cost" is a bit more folder clutter in S3. References • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md
a
@Rocky does run with docs.starrocks.io/docs/…/CREATE_STORAGE_VOLUME 
num_partitioned_prefix
helps to prevent S3 throttling? or file bundling?
r
The 
num_partitioned_prefix
setting is specifically designed to prevent S3 throttling, while a separate feature called File Bundling handles the issue of small file overhead. 1. 
num_partitioned_prefix
(Prevents S3 Throttling) This property, configured at the Storage Volume level, is used to scale S3 request throughput. * Mechanism: S3 enforces request rate limits per prefix (3,500 PUT/DELETE and 5,500 GET requests per second). By increasing 
num_partitioned_prefix
(default 256), StarRocks spreads data across more sub-paths (e.g., 
<s3://bucket/00/>
, 
<s3://bucket/01/>
...). * Purpose: It ensures that a high-concurrency cluster (with many parallel writes or reads) doesn't hit the 503 "Slow Down" errors from S3 by effectively multiplying the total allowed throughput of the bucket. 2. File Bundling (Prevents "Small File" Issues) If you are concerned about "file bundling" (combining many small files into fewer larger ones to reduce API costs), that is a different feature introduced in StarRocks v4.0. * Property: 
"file_bundling" = "true"
(configured in 
CREATE TABLE
properties). * Mechanism: It automatically bundles data files generated during loading, compaction, or publish operations. * Purpose: It reduces the API cost and metadata management overhead caused by high-frequency access to thousands of small files in external storage. Summary Comparison Feature | Where to Configure | Main Purpose 
num_partitioned_prefix
 | 
CREATE STORAGE VOLUME
| Prevents Throttling: Increases total IOPS/TPS by spreading data across more S3 prefixes. 
file_bundling
 | 
CREATE TABLE
(v4.0+) | Reduces API Costs: Combines small data files into larger bundles to lower object storage fees and metadata pressure. Recommendation: * If you are seeing 503 Slow Down errors, use a higher 
num_partitioned_prefix
. * If you are worried about high S3 API costs or too many small files, enable 
file_bundling
on your tables (available in v4.0+). References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • deployment/shared_data/s3.md References • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • deployment/shared_data/hdfs.md • deployment/shared_data/s3.md
a
@Rocky does starrocks support TrinoSQL, MySQL, and SparkSQL?
r
StarRocks supports Trino, MySQL, and Spark through different mechanisms: *1. MySQL (Native Protocol Compatibility)* StarRocks is protocol-compatible with MySQL. * Clients: You can connect to StarRocks using any standard MySQL client (like the MySQL CLI, DBeaver, or Navicat). * Syntax: It supports most MySQL-compliant SQL syntax and functions. * Integration: You can also query external MySQL databases directly from StarRocks using the MySQL JDBC Catalog. *2. Trino (Dialect & Translation)* StarRocks provides deep support for Trino users to simplify migrations or cross-system usage: * Trino Dialect: You can run 
SET sql_dialect = 'trino';
to use Trino-specific functions and syntax (like 
JSON
functions or specific 
ARRAY
operations) directly within StarRocks. * Translation Tool: As of v3.3.9, StarRocks includes a 
TRANSLATE TRINO <sql_statement>
command to help convert Trino SQL into the equivalent StarRocks SQL. *3. Spark (Connector-based)* StarRocks does not have a "SparkSQL dialect" inside the database, but it supports Spark via the StarRocks Connector for Apache Spark: * Spark SQL: Within a Spark environment, you can use the connector to create a temporary view on a StarRocks table and run Spark SQL queries against it. * Data Ingestion/Extraction: The connector allows Spark to read from and write to StarRocks tables using DataFrames and RDDs. * Data Lake Analytics: StarRocks can also query data formats often managed by Spark (such as Parquet, ORC, Delta Lake, Hudi, and Iceberg) via its External Catalogs. Summary Table Feature | Support Type | Key Mechanism MySQL | Native | MySQL Protocol & JDBC Catalog Trino | Dialect / Syntax | 
SET sql_dialect = 'trino'
Spark | Integration | StarRocks Spark Connector References • sql-reference/sql-statements/TRANSLATE_TRINO.md • sql-reference/System_variable.md • unloading/Spark_connector.md • data_source/feature-support-data-lake-analytics.md • faq/Sql_faq.md • release_notes/release-3.3.md • loading/Spark-connector-starrocks.md • sql-reference/sql-statements/table_bucket_part_index/SELECT.md References • unloading/Spark_connector.md • loading/Spark-connector-starrocks.md
a
@Rocky What are the data type requirement for stream load merge_condition?
r
The 
merge_condition
 parameter in StarRocks Stream Load (and other load methods like Routine Load and Broker Load) is used for conditional updates in Primary Key tables. Its primary purpose is to ensure that "older" data does not overwrite "newer" data, especially in high-concurrency or out-of-order ingestion scenarios. Data Type Requirements While StarRocks documentation does not provide an exhaustive "allow-list" of every single type, the requirements are dictated by the underlying comparison logic (the "Greater Than or Equal To" rule): 1. Comparable Types: The column must be of a data type that supports logical comparison. The update only takes effect if: 
Source_Record_Value >= Destination_Record_Value
2. Supported Types (Commonly Used): * Integer types: 
TINYINT
, 
SMALLINT
, 
INT
, 
BIGINT
, 
LARGEINT
(e.g., a version number or sequence). * Date and Time types: 
DATE
, 
DATETIME
(the most common use case, using a "last updated" timestamp). * Floating point: 
FLOAT
, 
DOUBLE
, 
DECIMAL
. * String types: 
CHAR
, 
VARCHAR
(uses lexicographical comparison). 3. Unsupported / Restricted Types: * Semi-structured & Complex types: 
JSON
, 
ARRAY
, 
MAP
, 
STRUCT
, 
BITMAP
, and 
HLL
are not supported for 
merge_condition
because they do not have a standard "greater than" comparison logic suitable for versioning. * Primary Key Columns: The column specified in 
merge_condition
cannot be part of the table's Primary Key. Functional Requirements * Table Type: Only Primary Key tables support this feature. * Column Placement: The column must exist in the StarRocks table and be mapped correctly in the load job. * Logic: If the incoming record's value is lower than the existing value in the specified column, the entire row's update is ignored for that specific primary key. Example Usage If you have a column named 
updated_at
(DATETIME), you would add the following header to your Stream Load command:
Copy code
bash
-H "merge_condition: updated_at"
Pro-Tip: Partial Updates 
merge_condition
is frequently used in conjunction with Partial Updates (
partial_update: true
). This ensures that even if you are only updating a few columns, the update only happens if the source data is actually newer than what is already stored. References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • sql-reference/data-types/semi_structured/VARIANT.md • table_design/table_types/table_capabilities.md • knowledge/usage/best-practice.md • developers/type_checker_xml_configuration.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/maxcompute_catalog.md
a
@Rocky does current starrocks flink stream load support writing as a map field?
r
The current StarRocks Flink connector supports writing Flink 
MAP
fields into StarRocks, but it does so by serializing the map into a JSON string during the transfer. Depending on your StarRocks table schema, this data can be stored in two ways: 1. Storing in a Native 
MAP
Column (Recommended for StarRocks 3.1+) If your StarRocks table uses the native 
MAP
data type (e.g., 
MAP<STRING, INT>
), the Flink connector can load data into it. * Mechanism: The connector converts the Flink 
Map
object into a JSON-formatted string (e.g., 
{"key1": 1, "key2": 2}
). * Requirements: * StarRocks Version: v3.1.12, v3.2.5, or later is recommended for full compatibility. * Connector Version: v1.2.10 or later is required for native type mapping support. * Configuration: It is highly recommended to set 
'sink.properties.format' = 'json'
in your Flink DDL to ensure the JSON strings are parsed correctly into the native 
MAP
structure. 2. Storing in a 
JSON
or 
STRING
Column For older versions of StarRocks or different use cases, you can map the Flink 
MAP
field to a StarRocks 
JSON
or 
VARCHAR
column. The connector will store the entire map as a serialized JSON object. Data Type Mapping Table (Writing/Sink) Flink Data Type | StarRocks Target Type | Notes 
MAP<KT, VT>
 | 
MAP
, 
JSON
, or 
STRING
| Serialized as a JSON string during Stream Load. 
ARRAY<T>
 | 
ARRAY<T>
 | Natively supported. 
ROW<...>
 | 
JSON
or 
STRING
| Serialized as a JSON string. Example Flink DDL
Copy code
sql
CREATE TABLE flink_sink (
    id INT,
    tags MAP<STRING, STRING> -- This will be sent as a JSON string
) WITH (
    'connector' = 'starrocks',
    'jdbc-url' = 'jdbc:<mysql://fe_ip:9030>',
    'load-url' = 'fe_ip:8030',
    'database-name' = 'test_db',
    'table-name' = 'test_table',
    'sink.properties.format' = 'json', -- Required for best results with complex types
    'username' = 'root',
    'password' = ''
);
Summary: While the "wire format" between Flink and StarRocks is a JSON string for map data, it is fully compatible with StarRocks native 
MAP
columns in recent versions. References • loading/Flink-connector-starrocks.md • unloading/Flink_connector.md • sql-reference/data-types/semi_structured/Map.md • release_notes/release-3.1.md References • loading/Flink-connector-starrocks.md • unloading/Flink_connector.md • sql-reference/data-types/semi_structured/Map.md • release_notes/release-3.1.md References • loading/Flink-connector-starrocks.md • unloading/Flink_connector.md • sql-reference/data-types/semi_structured/Map.md
a
@Rocky is 
combine_txn_log
tuning specific for shared-data?
r
Yes, the tuning parameter you are referring to, 
lake_use_combined_txn_log
, is specific to shared-data (compute-storage separation) architectures. What it is In StarRocks shared-data clusters, metadata and transaction logs are typically stored in remote object storage (like S3 or OSS). * Without it: Each transaction for a "Lake table" creates separate transaction log files. At high ingestion frequencies, this can lead to a large number of small files, increasing IOPS and cloud storage overhead. * With it: When 
lake_use_combined_txn_log
is set to 
true
, the system combines transaction logs for relevant transactions. This reduces the number of small files and improves the efficiency of transaction management in the shared-storage environment. Configuration Details * Parameter Name: 
lake_use_combined_txn_log
(FE configuration) * Default: 
false
* Compatibility: Available for shared-data clusters only (introduced in versions 3.3.7, 3.4.0, and 3.5.0). * Type: Boolean, Mutable (can be changed via 
ADMIN SET FRONTEND CONFIG
). Why it doesn't apply to Shared-nothing In a shared-nothing (classic) architecture, transaction logs are managed locally on the BE/FE nodes' disks using a different mechanism that doesn't involve the high latency/IOPS costs associated with object storage metadata operations. References • administration/management/FE_configuration.md • quick_start/shared-data.md • administration/management/compaction.md References • administration/management/FE_configuration.md
a
@Rocky Is there any downsize / existing open bug about enabling 
lake_enable_batch_publish_multi_table
 in shared-data?
r
In StarRocks shared-data (compute-storage separation) mode, 
lake_enable_batch_publish_multi_table
 is a performance-oriented configuration used to group transactions spanning different tables into a single publish operation. While it significantly improves throughput for high-frequency, multi-table ingestion (like CDC pipelines), there are specific "downsides" and historical bugs you should be aware of: 1. Known Issues & Fixed Bugs * Version Visibility Error ("The specified key does not exist"): In earlier implementations of batch publish, a query could occasionally request an intermediate transaction version that didn't have its own metadata object because the system jumped straight to the final version of the batch. This was addressed in recent versions (e.g., fixed via #79175), ensuring partitions advance correctly. * Deadlocks: There was a known issue where partition version gaps could cause a "Batch publish deadlock" (#71483). This has been addressed in version 3.3.8+ and 3.5.x. * Version Inconsistency: Some versions of StarRocks 3.3 (prior to 3.3.8) had an issue where batch publish could lead to version inconsistencies in metadata. 2. General Downsides / Risks * Increased Visibility Latency: Because transactions wait to be grouped into a batch, the time between a "commit" and the data becoming "visible" to queries (the 
commit-to-visible
latency) can increase slightly for the first few transactions in a batch. * Larger Rollback Surface: If a batch publish fails, all transactions within that batch are delayed or rolled back together. This increases the scope of a single point of failure compared to per-transaction publishing. * Rolling Upgrade Hazard: You must not enable this parameter until all FE nodes in your cluster have been upgraded to a version that supports the feature. Enabling it during a rolling upgrade can cause metadata corruption or FE crashes because older FEs won't understand the batched transaction logs. 3. Usage Recommendations * Default: 
false
. * Prerequisite: It only takes effect if 
lake_enable_batch_publish_version
is also set to 
true
. * Best Use Case: High-concurrency CDC workloads where many small transactions are hitting multiple tables. If you are doing large-scale batch loads, the benefits are negligible, and keeping it 
false
is safer. * Recommended Version: Ensure you are on at least v3.3.8, v3.4.1, or v3.5.x before enabling this to ensure you have the latest stability fixes for the batch publish logic. If you encounter issues after enabling it, check your FE leader logs for "transaction log list does not contain txn log" or "prev publish version task has not finished" errors. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • sql-reference/System_variable.md
a
@Rocky can starrocks support minute level freshness on materialized view? can starrocks support incremental refresh on materialized view?
r
Yes, StarRocks supports both minute-level freshness and incremental refresh for materialized views, though the mechanism for incremental refresh depends on your data source and StarRocks version. 1. Minute-Level Freshness You can achieve minute-level freshness using Asynchronous Materialized Views. There are two main ways to configure this: * Scheduled Refresh: You can specify a precise interval. For example, to refresh every minute:
Copy code
sql
    CREATE MATERIALIZED VIEW my_mv
    REFRESH ASYNC EVERY (INTERVAL 1 MINUTE)
    AS SELECT ...
* Automatic Refresh: If you use 
REFRESH ASYNC
(without the 
EVERY
clause), StarRocks will automatically trigger a refresh task as soon as it detects a data change in the base tables. This can provide even higher freshness than a fixed 1-minute interval. ────────── 2. Incremental Refresh StarRocks supports incremental refresh at different levels: A. Partition-Level Incremental Refresh (Standard) This is the most common method and is supported for both StarRocks native (OLAP) tables and most external catalogs (Hive, Iceberg, Hudi, Paimon). * How it works: If you define the MV with a 
PARTITION BY
clause that matches the base table's partitioning, StarRocks will only refresh the specific partitions that have changed. * Benefit: Instead of rebuilding the entire MV, it only recomputes the "dirty" partitions, significantly saving resources. B. Row-Level Incremental Refresh (New 
refresh_mode
) In recent versions (starting with v3.3/v3.4), StarRocks introduced a specific property to enable finer-grained incremental updates that process only append-only changes without re-scanning entire partitions. * Property: 
"refresh_mode" = "INCREMENTAL"
* Current Support: Primarily available for Iceberg and Paimon tables. * Operator Constraints: * Supported: 
Inner Join
, 
Aggregate with GROUP BY
(without 
DISTINCT
), 
Filter
, 
Union All
. * Not Supported: 
Outer Joins
or aggregations without a 
GROUP BY
clause. Summary Table Feature | Supported | Notes Minute Freshness | Yes | Use 
REFRESH ASYNC EVERY (INTERVAL 1 MINUTE)
. Partition Incremental | Yes | Works for OLAP & most Data Lake tables via 
PARTITION BY
. Row Incremental | Partial | Use 
"refresh_mode" = "INCREMENTAL"
(Iceberg/Paimon). References • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md • using_starrocks/async_mv/Materialized_view.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md References • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md • using_starrocks/async_mv/Materialized_view.md
a
@Rocky in starrocks 4.1.6 what is the algorithm slow down ingestion upon bad CS?
r
In StarRocks 4.1.6 (and generally since v3.2 in shared-data mode), the system uses a specific throttling algorithm to slow down ingestion when a partition's Compaction Score (CS) becomes too high ("bad CS"). This is designed to prevent "version explosion" (too many unmerged data files) which would otherwise degrade query performance and FE metadata stability. The Slowdown Algorithm When the 
lake_enable_ingest_slowdown
parameter is set to 
true
, the system calculates a delay for the transaction commit phase based on the following formula: Delay Time = 
Ingestion Task Duration
× (
Current CS
- 
lake_ingest_slowdown_threshold
) × 
lake_ingest_slowdown_ratio
 Key Parameters (FE Configuration) * 
lake_ingest_slowdown_threshold
: The CS value at which throttling begins. _ _Default:* 100. * 
lake_ingest_slowdown_ratio
: The multiplier that determines how aggressively the system adds delay. _ _Default:* 0.1. * 
lake_compaction_score_upper_bound
: The hard limit. If the CS reaches this level, new ingestion transactions for that partition are rejected entirely (returning an error) until compaction catches up. _ _Default:* 2000. Example Scenario If a data load task takes 5 minutes to write data, the 
lake_ingest_slowdown_ratio
is 0.1, and the partition's Compaction Score is 110 (which is 10 points above the threshold): * Calculation: $5\text{ mins} \times (110 - 100) \times 0.1 = 5\text{ minutes of delay}$. * Result: The total time for the load to complete will double to 10 minutes (5 mins writing + 5 mins artificial delay), effectively halving the ingestion throughput for that partition. Context for StarRocks 4.1.x While the slowdown algorithm remains consistent, version 4.1 introduced Adaptive Layout (specifically for shared-data) to mitigate the "bad CS" problem at the root. It automatically splits and merges tablets as data evolves, which helps maintain healthier tablet sizes and reduces the frequency of extreme compaction scores that trigger this slowdown. References • administration/management/FE_configuration.md • administration/management/compaction.md • best_practices/primarykey_table.md References • administration/management/FE_configuration.md • best_practices/primarykey_table.md • administration/management/BE_configuration.md • administration/management/compaction.md
a
@Rocky can we change max_varchar_length in cluster realtime? what is the real bottleneck?
r
In StarRocks, there isn't a single "real-time" toggle for the 
max_varchar_length
limit itself, as it is tied to the internal storage architecture. However, understanding the difference between the configuration and the architectural bottlenecks is key. 1. Can we change it in real-time? * The Hard Limit: For internal OLAP tables, the maximum length of a 
VARCHAR
is 1,048,576 bytes (1 MB). This is a hard architectural limit in the storage engine (since v2.1) and cannot be increased via configuration. * The Config (
max_varchar_type_length
): In some versions, an FE configuration 
max_varchar_type_length
exists to define the default length of the 
STRING
type (which is an alias). This is a *static parameter*; it requires a restart of the FE and modification of 
fe.conf
to take effect. * Protocol Limit: If you are hitting errors sending large strings from a client, you can change the 
max_allowed_packet
(default 32MB). This can be changed in real-time using 
SET GLOBAL max_allowed_packet = ...
. 2. What is the real bottleneck? Increasing 
VARCHAR
length to the 1MB limit for many columns or using the 
STRING
type (which defaults to a large length) introduces several performance bottlenecks: A. Memory Pre-allocation (The "Real" Bottleneck) This is the most critical issue. While StarRocks stores 
VARCHAR
as variable-length on disk, the execution engine (during Joins, Shuffles, and Aggregations) often pre-allocates memory based on the declared maximum length rather than the actual data size. * Impact: If you define a column as 
VARCHAR(1000000)
but only store 10-byte strings, the system may still reserve excessive memory during query processing. This leads to high memory pressure, lower concurrency, and frequent OOM (Out of Memory) errors. B. Total Row Size Limit The absolute limit for a single row in a StarRocks OLAP table is approximately 1,048,578 bytes. * Impact: If you have one 
VARCHAR(1MB)
column, you effectively cannot have any other significant columns in that table. If the sum of all column definitions in a row exceeds this limit, the 
CREATE TABLE
or 
ALTER TABLE
statement will fail. C. Short-Key Index Truncation StarRocks builds a sparse "short-key index" using the first 36 bytes of the sort keys. * Impact: A 
VARCHAR
column will truncate the short-key index. If you place a large 
VARCHAR
column early in your sort key, it reduces the effectiveness of the prefix index for all subsequent columns, slowing down range scans. D. Metadata Overhead Large 
VARCHAR
definitions increase the size of the FE metadata. If a cluster has thousands of tablets with many large 
VARCHAR
columns, it can lead to slower FE startup times and increased memory usage in the FE. Recommendation * Be Precise: Always set the 
VARCHAR
length to the minimum necessary value (e.g., 
VARCHAR(255)
instead of 
STRING
). * Storage Alternatives: If you truly need to store data larger than 1MB (like large JSON blobs or documents), consider storing them in an external system and keeping a URL/ID in StarRocks, or using JSON types (which also have a 1MB limit but are optimized for semi-structured access). References • sql-reference/data-types/string-type/VARCHAR.md • faq/table_design_faq.md • sql-reference/System_limit.md References • administration/management/FE_configuration.md • faq/Sql_faq.md
a
@Rocky how to check historical query history in starrocks cluster?
r
To check historical query history in a StarRocks cluster, you can use several methods depending on how long ago the queries ran and whether you have additional plugins installed. 1. The Most Direct Way: 
SHOW PROFILELIST
 This command is the quickest way to see a summary of recently executed queries (successful, failed, and long-running) directly from the SQL interface. * Command: 
SHOW PROFILELIST [LIMIT <num>];
* What it shows: Query ID, Start Time, Total Time, State, and the SQL statement. * Usage: Once you find a specific 
QueryId
, you can run 
ANALYZE PROFILE FROM '<QueryID>';
to get a deep dive into its execution plan and performance. 2. Using the Audit Log (Persistent History) StarRocks records every SQL request in an audit log on the Frontend (FE) nodes. * File Path: 
fe/log/fe.audit.log
* Contents: It includes the user, client IP, execution time, CPU/memory cost, and the full SQL text. * AuditLoader Plugin: For easier querying, most users install the AuditLoader plugin. This automatically loads the 
fe.audit.log
data into a StarRocks table (typically 
starrocks_audit_db*.starrocks_audit_tbl*
), allowing you to use standard SQL to analyze history:
Copy code
sql
    SELECT * FROM starrocks_audit_db*.starrocks_audit_tbl*
    WHERE queryTime > 1000 -- find queries taking > 1s
    ORDER BY timestamp DESC;
3. The 
_statistics_.query_history
Table Starting with version 3.2, StarRocks introduced a system table to store query history, primarily to support the SQL Plan Manager (SPM). * Prerequisite: You must enable it globally:
Copy code
sql
    SET GLOBAL enable_query_history = true;
* Querying:
Copy code
sql
    SELECT * FROM _statistics_.query_history
    WHERE dt >= NOW() - INTERVAL 1 DAY;
* Retention: Controlled by 
query_history_keep_seconds
(default is 3 days). 4. 
information_schema.warehouse_queries
(Shared-Data Mode) If you are using StarRocks in shared-data (cloud-native) mode with multiple warehouses, you can query this view to see history across specific compute resources. * Querying:
Copy code
sql
    SELECT * FROM information_schema.warehouse_queries
    WHERE WAREHOUSE_NAME = 'my_warehouse';
5. StarRocks Manager (GUI) If you have StarRocks Manager (the official web-based management platform) installed, there is a dedicated Query Analysis or Query History page. This provides a visual dashboard to filter, search, and analyze historical query profiles without using the command line. Summary of methods Method | Best For | Retention 
SHOW PROFILELIST
 | Quick debugging of recent queries | Limited (memory-based) Audit Log / AuditLoader | Long-term auditing and reporting | Configurable (disk/table-based) 
query_history
table | Plan analysis and performance tracking | Default 3 days StarRocks Manager | Visualizing and comparing queries | Depends on Manager config References • using_starrocks/running_queries.md • best_practices/query_tuning/query_profile_text_based_analysis.md • using_starrocks/SQL_plan_manager.md • administration/management/audit_loader.md References • sql-reference/sql-statements/table_bucket_part_index/SELECT.md • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md • sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW.md
2 Views
Open in Slack
PreviousNext
Powered by