http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> What does enable_lake_compaction_us...
# questions-and-troubleshooting
m
@Rocky What does enable_lake_compaction_use_partial_segments do? I cant find it in the documentation. 2. Speed up the oversized base compactions - enable partial-segment compaction. Your profiles show a real inefficiency on the hot partition. One base compaction (txn 12369) read 61 GB / 8,325 segments, sat in queue 1,254s (21 min), and wrote 43 GB over 22 min - and while it ran, other compactions on that partition queued behind it (in_queue_sec 126-188s on several). You have enable_lake_compaction_use_partial_segments = false. Turning it on lets base compaction rewrite far less data, shrinking those monster compactions and the queueing they cause: Was the response from claude with starrocks-debug skill
r
The parameter 
enable_lake_compaction_use_partial_segments
is currently not documented in the official StarRocks public documentation. It appears to be an internal or experimental configuration specifically for shared-data (lake) clusters that was likely introduced in very recent versions (v3.3 or later) or exists as a developer-level tuning knob. What it does (based on the context provided) Based on the description you found, this parameter changes how Base Compaction handles large data volumes in shared-data mode: * Standard Behavior (
false
): When a Base Compaction occurs, StarRocks typically reads the entire "Base" rowset plus all "Cumulative" rowsets and merges them into a single new rowset. For a large table, this means reading and rewriting tens or hundreds of gigabytes (e.g., your 61 GB example) even if the new data being added is relatively small. * Optimized Behavior (
true
): Turning this on enables partial-segment compaction. This allows the compaction engine to identify and rewrite only the specific segments (data files) that need to be merged or updated, rather than rewriting the entire multi-gigabyte rowset. * Result: It significantly reduces I/O and CPU usage during base compactions, prevents "monster" tasks from clogging the compaction queue, and reduces the "write amplification" seen on hot partitions. Related documented parameters While that specific parameter is hidden, there are several related configurations for tuning Lake Compaction in recent versions: 1. 
lake_enable_vertical_compaction_fill_data_cache
(CN/BE): Controls whether vertical compaction (which reduces memory by merging column-by-column) uses local disk cache to speed up the process. 2. 
lake_compaction_max_tasks
(FE): Limits the total concurrent compaction tasks in the cluster. 3. 
lake_compaction_score_selector_min_score
(FE): The threshold (default 10.0) that triggers compaction. 4. 
lake_pk_compaction_max_input_rowsets
(CN/BE): Limits how many files are merged at once for Primary Key tables to prevent long-running tasks. Recommendation If you are seeing the "monster compaction" behavior (high 
in_queue_sec
and massive read/write volumes for a single transaction), you can try setting this in the FE configuration:
Copy code
sql
-- To check if the config exists in your version
ADMIN SHOW FRONTEND CONFIG LIKE "%lake_compaction_use_partial_segments%";

-- To enable it (if it is a FE config)
ADMIN SET FRONTEND CONFIG ("enable_lake_compaction_use_partial_segments" = "true");
Note: Because this is undocumented, it may be experimental. It is highly recommended to test this in a non-production environment first or reach out to the StarRocks community Slack to confirm the stability of this feature for your specific version. References • administration/management/compaction.md • administration/management/BE_configuration.md • faq/shared_data_faq.md • administration/management/FE_configuration.md
m
@Rocky sum(rate(lake_publish_tablet_version_count[$interval])) is the rowset commit rate? So lower number means better client side batching and less compaction work needed? I have 144 compaction slots available (12 CN nodes) but only ~17-20 compaction tasks running concurrently. THink this is due to hot partition of current hour being written to, so I cant keep throwing more resource to it anymore. Will increasing merge commit interval help meaningfully too?
r
Yes, your understanding of both the metric and the current bottleneck is correct. 1. The Metric: 
lake_publish_tablet_version_count
 This metric tracks the number of times tablet versions are incremented due to a successful Publish Version (the final stage of a load transaction). * Is it the rowset commit rate? Yes, essentially. In StarRocks shared-data (Lake) mode, each transaction commit creates new rowsets. When these versions are published, this counter increments. * Lower is better: A lower rate indicates that each transaction is carrying more data (better batching). High rates mean many small files are being created, which is the primary driver of "Compaction Debt" and high Compaction Scores. 2. The Compaction Parallelism Bottleneck You have 144 slots (likely 
compact_threads
× CN nodes), but only 17-20 tasks are running. This is a common behavior when dealing with hot partitions: * Partition-Level Scheduling: StarRocks FE schedules compaction tasks based on partitions. If only a few partitions (e.g., the current hour) have high scores, only those partitions are eligible for compaction. * Serial Limitations: Even if a partition has a huge backlog, StarRocks generally limits the number of concurrent compaction tasks per partition or tablet to maintain version consistency and prevent I/O contention on specific data files. * The Result: You cannot "brute force" a single hot partition's compaction by adding more CN nodes or global threads once you hit these per-partition limits. 3. Will increasing Merge Commit Interval help? Yes, significantly. Increasing the 
merge_commit_interval_ms
(used in Stream Load or the Flink Connector) is one of the most effective ways to alleviate this: * How it works: It allows StarRocks to buffer multiple incoming stream load requests and commit them as a single transaction. * Impact: Instead of 100 transactions creating 100 small rowsets, you get 1 transaction creating 1 larger rowset. This directly reduces the 
lake_publish_tablet_version_count
and prevents the "small file problem" before it even reaches the compaction engine. * Recommendation: If you are currently at 1s or 5s, try increasing it to 10s or 15s. This will reduce the number of files the compaction engine has to "look at" and merge, making the tasks it does run much more efficient. Summary of Recommendations 1. Reduce Ingestion Frequency: Increase 
merge_commit_interval_ms
in your Flink/Stream Load configuration. 2. Use Partial Compaction: Enable 
enable_lake_compaction_use_partial_segments = true
(if available in your version) to stop the "monster" base compactions from rewriting the entire partition when only a few segments need merging. 3. Tune Cumulative Compaction: You can also reduce 
max_cumulative_compaction_num_singleton_deltas
to 
100
(from the default 500/1000) on the CNs. This makes individual compaction tasks finish faster, which helps clear the queue on hot partitions more quickly. References • administration/management/compaction.md • loading/Flink-connector-starrocks.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • administration/management/FE_configuration.md References • loading/Flink-connector-starrocks.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • knowledge/admin_and_operation/shared_data_compaction.md
m
@Rocky But does max_cumulative_compaction_num_singleton_deltas apply in primary key tables?
r
No, 
max_cumulative_compaction_num_singleton_deltas
does not apply to Primary Key (PK) tables. Primary Key tables use a different compaction mechanism called Update Compaction, which has its own specific parameters depending on your cluster type (Shared-data/Lake vs. Shared-nothing). 1. If you are on Shared-data (Lake) Cluster For PK tables in Lake mode, the parameter you should use is: * 
lake_pk_compaction_max_input_rowsets
 (CN config) * What it does: Limits the number of rowsets merged in a single PK compaction task. * Recommendation: In v3.3+, the default is 
500
. If you have very frequent small writes and want to speed up individual tasks to clear the queue, you can reduce this (e.g., to 
100
), but be aware this may increase the number of total compaction rounds needed. 2. If you are on Shared-nothing (Classic) Cluster For PK tables in classic mode, the parameter is: * 
max_update_compaction_num_singleton_deltas
 (BE config) * What it does: The equivalent of the "cumulative" limit but specifically for the Update Compaction process used by PK tables. * Default: 
1000
. Why the distinction matters * Cumulative Compaction (controlled by 
max_cumulative_compaction_num_singleton_deltas
) is designed for Duplicate, Aggregate, and Unique (Merge-on-Read) models. It merges base data with new deltas. * Update Compaction is designed for the Primary Key (Update-on-Write) model. It not only merges data but also handles the deletion of old versions and the updating of the Primary Key index and delete vectors. Because this process is more memory-intensive, it uses dedicated "Update" configuration knobs. Summary of relevant PK Compaction configs: Config | Cluster Type | Description 
lake_pk_compaction_max_input_rowsets
 | Shared-data | Max rowsets per PK compaction task. 
max_update_compaction_num_singleton_deltas
 | Shared-nothing | Max rowsets per PK compaction task. 
lake_compaction_score_selector_min_score
 | Shared-data | Threshold to start compaction (default 10). 
update_compaction_per_tablet_min_interval_seconds
 | Shared-nothing | Minimum time between compactions for a single PK tablet. Recommendation: For your "hot partition" issue, stick with increasing the merge commit interval on the client side and using 
lake_pk_compaction_max_input_rowsets
 on the CNs to ensure tasks finish fast enough to keep the queue moving. References • administration/management/BE_configuration.md • administration/management/compaction.md • best_practices/primarykey_table.md • knowledge/admin_and_operation/shared_data_compaction.md References • administration/management/BE_configuration.md • administration/management/compaction.md • best_practices/primarykey_table.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md
m
@Rocky So reducing 
lake_pk_compaction_max_input_rowsets
 should be fine to reduce rowsets per tablet so queries are faster? My CPUs currently under utilized anyway. Any caveats?
r
Reducing 
lake_pk_compaction_max_input_rowsets
 (e.g., from 500 down to 100) is a common and often recommended tuning step for workloads with frequent small writes. Since your CPUs are underutilized, this is a very safe "lever" to pull. Why it helps * Faster Task Completion: A compaction task merging 100 files finishes much faster than one merging 500. This clears the "hot partition" queue more quickly. * Query Performance: Queries on Primary Key tables must merge data from all uncompacted rowsets at read time. Reducing the number of rowsets per tablet (by ensuring compaction keeps up) directly lowers the 
SegmentsReadCount
and improves query latency. * Resource Smoothing: Instead of one "monster" compaction task that hogged a thread for 20 minutes (as seen in your profile), you get smaller, bite-sized tasks that better utilize your 144 available slots. The Caveats While generally beneficial, there are a few things to keep in mind: 1. Write Amplification: If you set this number too low, you might merge the same data multiple times as it moves through different "levels" of the compaction hierarchy. However, at 100, this is rarely an issue for StarRocks. 2. Size-Tiered Strategy: In StarRocks v3.2+, there is a new strategy called Size-tiered Compaction for PK tables (enabled by 
enable_pk_size_tiered_compaction_strategy = true
). If this is ON, the system automatically groups rowsets of similar sizes. In this mode, 
lake_pk_compaction_max_input_rowsets
acts more as a "safety cap" rather than the primary driver of task size. 3. FE Scheduling Latency: Each compaction task requires a tiny bit of coordination from the FE. If you have millions of tablets and very small tasks, the FE might spend more time scheduling than the CNs spend executing. With only 17-20 concurrent tasks, you are nowhere near this limit. 4. I/O Cost: Each task involves reading and writing to object storage (S3/GCS/OSS). More tasks mean more API calls to the storage provider, which can marginally increase costs, though this is usually negligible compared to the performance gains. Recommendation Yes, go ahead and reduce it to 100 or 200.
Copy code
sql
-- On a CN node (or via information_schema for all CNs)
UPDATE information_schema.be_configs 
SET VALUE = "100" 
WHERE name = "lake_pk_compaction_max_input_rowsets";
Check the result: Monitor your Compaction Score and 
SegmentsReadCount
 in your query profiles. If the score stays low and your queries speed up without hitting CPU bottlenecks, you’ve found a sweet spot. Combined with increasing your merge commit interval (to produce fewer, larger rowsets in the first place), this should solve the "monster compaction" and queueing issues you are seeing. References • administration/management/compaction.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md • knowledge/admin_and_operation/shared_data_compaction.md References • administration/management/BE_configuration.md • administration/management/compaction.md • best_practices/primarykey_table.md • knowledge/admin_and_operation/shared_data_compaction.md
m
@Rocky im getting bandwidth out allowance exceeded on a lot of my i4i.8xlarge nodes for starrocks shared data primary key table. WHat can I do to remediate this? Aside from switching to a different machine family. If I need to change machines, what type of machines should I use
r
The "Bandwidth out allowance exceeded" error on AWS 
i4i.8xlarge
nodes indicates that your StarRocks Compute Nodes (CNs) are hitting the network throughput limits imposed by the AWS Nitro system. In a Shared-data (Lake) Primary Key environment, "Bandwidth Out" is consumed primarily by two activities: 1. Writing to S3 during Ingestion: Every 
merge_commit
writes new data files (segments) to S3. 2. Writing to S3 during Compaction: This is the most likely culprit. Compaction reads existing data from S3, merges it, and uploads the new merged segments back to S3. 1. Remediation (Configuration Changes) Since you are hitting a bandwidth cap, the goal is to "smooth out" the network spikes and reduce the total volume of data being uploaded. * Throttle Compaction Concurrency (FE): You currently have ~144-192 potential slots. Even if only 20 tasks are running, if they are all uploading large files simultaneously, they can easily saturate the 25 Gbps burst limit of an 
i4i.8xlarge
.
Copy code
sql
    -- Reduce the total number of concurrent compaction tasks across the whole cluster.
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "48"); -- Try 4 per node instead of the default 16.
* Limit Threads per CN (CN): Lowering the number of threads that can perform I/O-intensive work on each node:
Copy code
sql
    UPDATE information_schema.be_configs SET VALUE = "4" WHERE name = "compact_threads";
* Increase `merge_commit_interval_ms`: As we discussed, if you commit every 1 second, you upload many small files. If you commit every 10 seconds, you upload fewer, larger files. This reduces the metadata overhead and total write requests to S3, which helps with overall bandwidth stability. * Increase Compression: If you are using the default (LZ4), switching to ZSTD for your table can significantly reduce the amount of data physically sent over the wire to S3, though it uses more CPU.
Copy code
sql
    -- For new partitions/data
    ALTER TABLE your_table SET ("compression" = "ZSTD");
2. If you must change Machine Types The 
i4i.8xlarge
provides "Up to 25 Gbps" network bandwidth. This is a burst limit. Once you exceed your "allowance" (token bucket), AWS throttles you to a much lower baseline (often 10-12 Gbps). Option A: Vertical Scaling within the same family (Recommended) If you like the 
i4i
family because of the local NVMe (which StarRocks uses for Data Cache), simply move to a larger size. Network bandwidth on AWS scales with instance size: * 
i4i.16xlarge
: 37.5 Gbps. * 
i4i.24xlarge
: 50 Gbps. * 
i4i.32xlarge
: 75 Gbps. Moving to 
i4i.16xlarge
gives you a significantly higher baseline and burst allowance. Option B: Network-Optimized Instances ("n" suffix) If your bottleneck is strictly network and you can sacrifice some local NVMe cache size per dollar: * 
r6in.8xlarge
or 
r7iz.8xlarge
: These "n" and "z" variants are designed for high-performance networking (up to 50 Gbps or 100 Gbps). * Caveat: Ensure they still have enough local NVMe (or very fast EBS) for the StarRocks Data Cache, otherwise, your query performance will tank due to S3 read latency. Option C: Graviton3 Instances (
is4gen
or 
im4gn
) If you are open to ARM-based instances (StarRocks supports Graviton): * 
is4gen
 instances have even higher networking throughput relative to their size compared to the 
i4i
series and are very cost-effective for StarRocks Shared-data workloads. Summary Recommendation 1. Immediate: Lower 
lake_compaction_max_tasks
to 48 (FE) and 
compact_threads
to 4 (CN). 2. Next Step: Increase your ingestion 
merge_commit_interval_ms
to at least 10s. 3. Hardware: If the above doesn't stop the throttling, upgrade your nodes to 
i4i.16xlarge
. The jump from 25 Gbps to 37.5 Gbps usually provides enough headroom to prevent Nitro throttling. References
m
@Rocky I currently have 12 i4i.8xlarge instances. Would changing cluster topology to 6 i4i.16xlarge help?
r
Yes, changing your cluster topology from 12 
i4i.8xlarge
 nodes to 6 
i4i.16xlarge
 nodes is a very effective way to remediate "Bandwidth out allowance exceeded" errors in StarRocks. While the total cluster CPU and Memory remain the same, the individual node ceilings change in a way that directly addresses Nitro throttling. Why consolidating to 
i4i.16xlarge
helps: 1. Higher Bandwidth Floor and Ceiling: * 
i4i.8xlarge
: Has a baseline of ~10 Gbps and a burst of 18.75 Gbps (some regions up to 25). * 
i4i.16xlarge
: Has a baseline of ~20 Gbps and a burst of 37.5 Gbps. Moving to the larger size doubles the "headroom" for any single heavy compaction task or data upload. Throttling often occurs because a single node is working on a particularly large tablet/partition; an 
8xlarge
will hit its cap much sooner than a 
16xlarge
. 2. More Robust "Token Bucket": AWS Nitro bandwidth is managed via a token bucket (credit) system. Larger instances have a larger bucket and a faster refill rate. By consolidating, you are effectively creating 6 "larger buckets" instead of 12 "smaller buckets," which makes the cluster much more resilient to spikes (micro-bursts) common in StarRocks compaction. 3. Reduced Intra-Cluster Overhead: In an MPP (Massively Parallel Processing) architecture like StarRocks, queries often involve shuffling data between nodes. With 6 nodes instead of 12, there are fewer "network hops" required for many query plans, reducing the total cluster-wide "Bandwidth Out" consumption for the same workload. Potential Caveats of Consolidation: * Blast Radius: If one 
16xlarge
node fails or needs maintenance, you lose 16.6% of your cluster capacity instead of 8.3%. * Data Cache Warming: After the migration, each of the 6 nodes will need to manage twice the data in its local NVMe cache. Ensure your 
datacache_mem_size
and disk space are scaled up proportionally (which happens naturally with 
i4i.16xlarge
as it has 4x 3.75TB NVMe drives). Alternative Machine Families If you decide to switch families entirely, look for the "n" (network-optimized) variants: * 
r6in.8xlarge
/ 
r7iz.8xlarge
: These are designed specifically for high network throughput (up to 50-100 Gbps). * 
is4gen.8xlarge
: These are AWS's newer Graviton3-based storage instances. They have very high networking performance relative to their cost and excellent NVMe storage for the StarRocks Data Cache. This is often the "Goldilocks" choice for StarRocks shared-data clusters on AWS. Summary Recommendation Change to 6 nodes of 
i4i.16xlarge
. It is the "path of least resistance" because it keeps you in the same hardware family (no ARM migration or different storage drivers) but effectively doubles the network throughput ceiling per node, which is the specific bottleneck causing your "allowance exceeded" errors. Action Item before switching: Verify your 
lake_compaction_max_tasks
 is not set too high. If you consolidate to 6 nodes but still allow 16 tasks per node (the default), you might still burst over the 
16xlarge
limit. Start with 8 tasks per node on the new 
16xlarge
instances. References • deployment/deploy_manually.md • administration/management/BE_configuration.md • administration/management/compaction.md • best_practices/primarykey_table.md • knowledge/usage/best-practice.md
m
@Rocky But is StarRocks compatible with ARM architecture? I thought only x84 works? So 6 is4gen.8xlarge is the best? Doesn't the total outgoing data to S3 remain the same though? So reducing to 6 nodes just compresses the amount of data going out from 12 to 6 nodes but same total outgoing bytes
r
Yes, StarRocks is fully compatible with ARM architecture. Since StarRocks v3.3, ARM (specifically AWS Graviton) has been treated as a "first-class citizen" with dedicated instruction-set optimizations. In fact, official benchmarks (SSB, Clickbench) show that StarRocks on Graviton3 can be 11% to 39% faster than the equivalent x86 architecture at a lower cost. 1. Is 6 
is4gen.8xlarge
the best? Not necessarily for your current workload. Here is why: * CPU Match: Your current cluster (12 
i4i.8xlarge
) has 384 vCPUs. Switching to only 6 
is4gen.8xlarge
would drop you to 192 vCPUs (a 50% reduction in compute). * Network Match: The 
is4gen.8xlarge
has a network burst of 25 Gbps, which is the same as your current 
i4i.8xlarge
. Switching to the same size in a different family won't solve the bandwidth throttling if the per-node ceiling remains the same. Recommendation: To solve the bandwidth issue while keeping your compute power, look at nodes that have a higher network ceiling, such as 
i4i.16xlarge
 (37.5 Gbps) or 
is4gen.16xlarge
 (50 Gbps). 2. The "Total Outgoing Data" Logic You are correct that the total bytes uploaded to S3 over an hour will be roughly the same. However, AWS throttles you based on micro-bursts at the individual node level. Consolidating from 12 nodes to 6 larger nodes helps because: * The Individual "Pipe" is Wider: AWS Nitro limits are per-instance. If a single 
i4i.8xlarge
node tries to upload 20 Gbps during a heavy compaction, it hits its limit and gets throttled. An 
i4i.16xlarge
node doing the same 20 Gbps upload is perfectly fine because its limit is 37.5 Gbps. * Handling Skew: Data is rarely perfectly balanced. Usually, one or two "hot" tablets trigger massive compaction tasks. In a 12-node cluster, those hot tasks are restricted to a 25 Gbps pipe. In a 6-node cluster of larger machines, those same hot tasks have a much higher ceiling to finish their burst without triggering the "allowance exceeded" error. * Token Bucket Refill: Larger instances have a larger "token bucket" for network credits and a faster refill rate. They can sustain high-bandwidth activity for longer periods than smaller instances. Summary Strategy 1. ARM is great: If you want to move to Graviton, it is supported and recommended for cost-performance. 2. Size matters for Network: To stop the "allowance exceeded" error, you need nodes with a higher network specification (look for 37.5 Gbps or 50 Gbps in the AWS console). 3. The Sweet Spot: Moving to 6x 
i4i.16xlarge
 (x86) or 6x 
is4gen.16xlarge
 (ARM) would maintain your 384 vCPU count but significantly increase your per-node network headroom, which is exactly what you need to stop the throttling. References • release_notes/release-3.3.md • deployment/deployment_prerequisites.md • developers/build-starrocks/build_starrocks_on_ubuntu.md • developers/build-starrocks/Build_in_docker.md • faq/Deploy_faq.md References • release_notes/release-3.3.md • deployment/deployment_prerequisites.md • developers/build-starrocks/build_starrocks_on_ubuntu.md • developers/build-starrocks/Build_in_docker.md • faq/Deploy_faq.md • deployment/plan_cluster.md
m
@Rocky But each micro burst at the node level will be bigger? Effectively assuming even data distribution, each micro burst twice as big?
r
This is a common and very sharp observation. You are essentially asking: "If I have half the nodes doing twice the work, doesn't the burst per node just double, hitting the new (larger) limit anyway?" The answer is no, and for three specific reasons related to how StarRocks schedules tasks and how AWS Nitro bandwidth works. 1. Task Size is Constant, Not Proportional In StarRocks, a "compaction task" happens at the tablet level. • A single tablet compaction task merges a specific number of rowsets (governed by 
lake_pk_compaction_max_input_rowsets
). • Moving to 6 nodes does not make a single tablet compaction task twice as large. The "burst" from one individual task remains identical in size (e.g., uploading 1GB of data). • The difference is that on an 
i4i.16xlarge
, that 1GB upload has a 37.5 Gbps pipe, whereas on the 
8xlarge
it only had 18-25 Gbps. It finishes faster and stays further away from the throttling ceiling. 2. Default Scheduling Throttles the Total Load StarRocks' adaptive scheduler (
lake_compaction_max_tasks = -1
) calculates the total cluster-wide compaction limit as 
Number of Nodes * 16
. • 12 Nodes: Total cluster allows 192 concurrent tasks. • 6 Nodes: Total cluster allows 96 concurrent tasks. By consolidating to 6 nodes, you are effectively telling the FE to be 50% more conservative by default. Even though each node is twice as powerful, the FE is sending fewer total concurrent tasks to the cluster. This naturally "smoothes" the network spikes that were previously saturating your 12 smaller nodes. 3. The "Token Bucket" is more than 2x more resilient AWS Nitro bandwidth limits aren't just "speed limits"; they are "credit limits" (Token Buckets). • The Refill Rate: A 
16xlarge
refills its "bandwidth tokens" significantly faster than an 
8xlarge
. • The Threshold: Throttling (the "Allowance Exceeded" error) triggers when your burst sustains long enough to empty your bucket. • Because the 
16xlarge
has a much higher baseline (guaranteed) bandwidth, you aren't dipping into your "burst credits" nearly as often. Most compaction tasks that would have exhausted the "allowance" of an 
8xlarge
node will fit entirely within the baseline throughput of a 
16xlarge
. Summary: Why 6 is better than 12 While the total data being moved to S3 is the same, the shape of that data movement changes: • Old (12 nodes): Many small "pipes" (18 Gbps) frequently hitting their ceiling and running out of credits. • New (6 nodes): Fewer, much wider "pipes" (37.5 Gbps) that can handle the exact same tablet-sized bursts without ever touching the "allowance" limit. Recommendation If you switch to 6x 
i4i.16xlarge
: 1. Keep 
compact_threads
at the default 4 or 8 initially. 2. Let StarRocks use the default 
lake_compaction_max_tasks = -1
. 3. You will likely see the "Bandwidth Out Allowance Exceeded" errors disappear because you've doubled your baseline (non-burstable) network floor. Is ARM still an option? Yes, and if you go with 6x 
is4gen.16xlarge
, you get 50 Gbps per node. That is a massive amount of headroom that almost guarantees you will never see a bandwidth throttling error again for this workload levels similar to what you have now. References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md
m
@Rocky Does compaction score matter if all my data lives on NVMe? Does the tablet rowset count affect queries?
r
Yes, the compaction score matters significantly, even if your data is cached on high-speed NVMe. While NVMe masks some of the pain of high rowset counts, it does not eliminate the fundamental overhead. Here is the breakdown of why compaction score and rowset counts are critical for query performance in StarRocks: 1. NVMe helps with I/O, but not "Read Amplification" NVMe improves I/O latency (how fast a single block is read), but it doesn't change the Read Amplification caused by uncompacted data: * Logical Overhead: If a tablet has a compaction score of 100 (meaning ~100 rowsets), the query engine must open 100 different files, read 100 sets of metadata (footers, bloom filters, zone maps), and manage 100 separate iterators in memory. * CPU Bottleneck: For Duplicate and Unique key models, the engine must perform a Priority Merge across all rowsets at read-time to find the correct version of a row. This is a CPU-intensive operation. NVMe makes the data available faster, but the CPU still has to "sort through the mess," which becomes a bottleneck as rowset counts grow. 2. The Impact on Primary Key Tables If you are using Primary Key tables (common in shared-data clusters), rowset count is even more vital: * Delete Vector Lookups: For every row read, StarRocks must check the Delete Vector to see if that row has been marked as deleted in a subsequent rowset. As the number of rowsets grows, the management and lookup of these vectors become more complex. * Metadata Memory: Each rowset consumes memory in the BE process for metadata. Too many rowsets can lead to "Metadata Bloat," increasing the memory footprint and potentially leading to OOM (Out of Memory) issues. 3. Compaction Score as a "Safety Valve" The compaction score isn't just a performance hint; it's a stability metric. StarRocks uses it to protect itself: * Ingest Throttling: When the 
max_compaction_score
of a partition exceeds 100, StarRocks begins to slow down (throttle) incoming write transactions. * Ingest Rejection: If the score reaches 2000, the cluster will reject new imports for that partition entirely to prevent the tablet from becoming unreadable. 4. Query Performance "Cliff" Queries usually don't degrade linearly with rowset counts; they hit a "cliff." * 0-20 Rowsets: Performance is near-optimal. * 50+ Rowsets: You will start seeing a noticeable increase in "Scan Time" and "CPU Time" in your Query Profile, even if the data is 100% cached in NVMe. * 100+ Rowsets: Small, point-lookup queries might remain fast, but large analytical scans will slow down significantly because the "Merge" phase of the scan becomes the dominant cost. Summary Recommendation * Don't rely on NVMe to save you: NVMe is a powerful tool for reducing S3 latency, but healthy compaction is what keeps your CPU cycles focused on processing data rather than merging files. * Monitor your scores: Keep your 
MaxCS
(Max Compaction Score) under 30 for optimal performance. You can check this with:
Copy code
sql
    SELECT * FROM information_schema.partitions_meta ORDER BY MAX_CS DESC;
* Tune if needed: If your scores are consistently high despite having enough hardware, consider increasing 
lake_compaction_max_tasks
(on FE) or reducing the frequency of very small, frequent loads (which create many tiny rowsets). References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md • faq/Deploy_faq.md References
m
@Rocky How does compaction work on the cache then? Since rowsets exist on the CN nodes and S3
r
In a StarRocks shared-data cluster (Compute-Storage Separation), compaction interacts with the cache in a way that prioritizes performance but still relies on S3 as the source of truth. Here is the step-by-step data flow for compaction and its relationship with the cache: 1. The "Read" Phase (Cache Hit vs. Miss) When a CN node is assigned a compaction task for a tablet, it needs to scan the existing rowsets (versions). * Cache Hit: If the data blocks for those rowsets are already in the local NVMe cache (e.g., they were recently loaded or queried), the CN reads them directly from the NVMe. This makes compaction very fast and saves S3 bandwidth. * Cache Miss: If the rowsets have been evicted from the cache or were never queried, the CN must download them from S3. As it downloads these blocks for compaction, it also populates them into the local cache (standard "Read-Through" behavior). 2. The "Write" Phase (Double Write) Compaction merges several small rowsets into one large rowset. * S3 Upload: The new, compacted rowset is uploaded to S3 to ensure data persistence and visibility for other CN nodes. * Cache Population: Similar to a standard data load, StarRocks typically writes the output of compaction to the local NVMe cache at the same time. This ensures that the new version is immediately "hot" and ready for high-speed queries without needing to be fetched from S3 again. 3. What happens to the "Old" Rowsets? Once a compaction transaction is successfully committed and published by the FE: * Invalidation: The metadata in the FE is updated to point to the new version. The old rowsets are marked for deletion (vacuuming). * Cache Cleanup: The blocks belonging to the old rowsets remain in the CN's local NVMe cache but are now "cold" because no new queries will request them. They aren't always deleted instantly; instead, they are naturally evicted by the LRU/SLRU policy as the new compacted data (and new ingests) require space. Why this matters for your Bandwidth/Throttling issue: If you have a high "Compaction Score," it means you have many small uncompacted files. * Heavy S3 Traffic: If your cache is small or your data turns over quickly, the CNs will constantly be downloading those small files from S3 to compact them, and then uploading the merged result back to S3. * The "Write" Burst: The final step of compaction (uploading the large merged file) is exactly what triggers the AWS "Bandwidth Out" throttling you are seeing. Pro-Tip: By moving to larger nodes (like the 
.16xlarge
mentioned before), you aren't just getting more network bandwidth; you are also getting a larger NVMe cache. A larger cache means more compaction tasks happen using local data (Cache Hits), which significantly reduces the Incoming bandwidth pressure on your nodes. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • data_source/data_cache.md • introduction/Architecture.md • deployment/shared_data/feature-support-shared-data.md References • knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/feature-support-shared-data.md • administration/management/compaction.md
m
@Rocky What happens if my CN nodes are exceeding their bandwidth by 40-100k packets per second? Do I get data loss?
r
No, you will not experience data loss due to network throttling or packet drops. However, you will experience cluster instability and transaction failures. StarRocks is designed with a "Single Source of Truth" (S3) and atomic commits. Here is exactly what happens when you exceed those packet-per-second (PPS) limits: 1. Why there is no data loss * Atomic Transactions: When you load data, the CN (Compute Node) uploads files to S3. Only after the S3 upload is successful does the CN notify the FE (Frontend) to "commit" the transaction. If the network drops packets and the S3 upload fails, the CN will report an error, and the FE will never mark that data as "visible." * Source of Truth: Because the metadata (managed by the FE) only points to successfully uploaded and committed files in S3, a network failure simply means the new data "never happened" from the database's perspective. You will need to retry the load from your source. 2. What actually happens (The "Chaos" Phase) While your data is safe, your cluster will become very difficult to use: * Connection Timeouts & RPC Errors: StarRocks nodes communicate via Internal RPC (brpc). If you are dropping 40k–100k packets per second, these internal "heartbeats" and "data shuffle" packets will fail. You will see errors like 
Internal Error: RPC standard request failed
or 
Master FE not found
. * CNs "Flapping" (Node Stability): The FE monitors CN health via heartbeats. If network congestion prevents these heartbeats from getting through, the FE will mark the CN as 
Dead
. The FE will then try to re-route all queries and compaction tasks to the remaining "Alive" nodes, which will likely push their network even further over the limit, causing a "cascading failure." * Compaction Backlog: As mentioned before, compaction requires uploading merged data to S3. If these uploads fail due to packet loss, your "Compaction Score" will skyrocket. This eventually leads to the cluster rejecting new writes (to prevent the tablet from becoming unreadable). 3. Why 100k PPS is a "Danger Zone" on AWS On AWS, "Allowance Exceeded" for PPS is often more disruptive than throughput throttling: * The Nitro Card: When the Nitro card hits its PPS limit, it doesn't just slow down; it begins tail-dropping packets. This causes TCP "Congestion Collapse," where the system spends more time re-transmitting old packets than sending new ones. * Small Packets: Since StarRocks uses many small RPC calls for metadata and coordination, you hit the PPS limit much faster than the GB/s limit. Recommendation If you are seeing 40k–100k packets dropped: 1. Consolidate to Larger Instances: As discussed, moving from 
8xlarge
to 
16xlarge
(or moving to 
is4gen
ARM instances) effectively doubles or triples your PPS allowance. 2. Check `brpc_connection_type`: Ensure your BE/CN config 
brpc_connection_type
is set to 
pooled
(the default in newer versions). This reduces the number of individual connections and packets required for node-to-node communication. 3. Batch Your Ingests: If you are doing thousands of tiny 
Stream Loads
, each one creates network overhead. Larger batches = fewer packets. Summary: Your data is safe in S3, but your cluster will essentially "stop breathing" until the network pressure is relieved. References • administration/management/compaction.md • introduction/Architecture.md • faq/Sql_faq.md • loading/loading_introduction/loading_considerations.md References • faq/Sql_faq.md • faq/Others.md • deployment/downgrade.md
m
@Rocky sum(rate(rpc_server_8060_starrocks_lake_service_aggregate_publish_version_sum{starrocks_cluster="$cluster"}[$interval])) / sum(rate(rpc_server_8060_starrocks_lake_service_aggregate_publish_version_count{starrocks_cluster="$cluster"}[$interval])) What does this represent. What units should it be in
r
This Prometheus query calculates the Average Latency of the 
aggregate_publish_version
RPC call within a StarRocks shared-data (Lake) cluster. 1. What does it represent? * Service (
starrocks_lake_service
): This metric belongs to the Lake Service, which is the internal component responsible for managing storage and metadata in StarRocks' shared-data (compute-storage separation) architecture. * Operation (
aggregate_publish_version
): This is a critical stage in the data ingestion lifecycle. After data has been written to S3/object storage, the Frontend (FE) sends this RPC to the Compute Nodes (CNs) or BEs to "publish" the new version. This step updates the local metadata to make the new data visible for queries. The "aggregate" prefix indicates that it is a batched operation, publishing multiple versions or tablets in a single call for efficiency. * The Calculation: By dividing the rate of the sum of durations (
_sum
) by the rate of the count of requests (
_count
), you are deriving the average time each request took to complete over the specified 
$interval
. 2. What units is it in? The units are Microseconds (μs). StarRocks uses the 
brpc
(Apache bRPC) framework for internal communication. By default, 
brpc
exposes its latency metrics in microseconds. * Numerator (
_sum
): Total microseconds spent processing these requests. * Denominator (
_count
): Total number of requests processed. * Result: Average microseconds per request. Why this metric matters: If you see this value spiking: * Metadata Bottleneck: It often indicates that the CN/BE is struggling to update its internal tablet metadata or that there is a contention in the metadata lock. * Ingestion Latency: Since this is the final step of a transaction, high latency here directly increases the "Commit" time of your 
Stream Load
or 
INSERT
jobs. * Network/RPC Pressure: High latency could also be a symptom of the "Packet per Second" (PPS) throttling mentioned in your previous questions, as the RPC responses themselves may be delayed or retried due to packet loss. Common Health Check: For a healthy cluster, this value is usually under 10,000μs (10ms). If it consistently exceeds 100ms (100,000μs), it indicates a significant delay in making new data available for querying. References • using_starrocks/sorted_aggregate.md • table_design/table_types/aggregate_table.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • unloading/arrow_flight.md
m
@Rocky I changed my cluster topology from 12 i4i.8xlarge machines to 6 i4i.16xlarge machines since I was having bandwidth exceed allowance issues. Even with bandwidth allowance exceeded problems gone now, query times are much worse. Should I choose 24 i4i.4xlarge nodes as my cluster topology instead? It has up to 25 Gibps network bandwidth while i4i.8xlarge has 18.75 Gibps only
r
The short answer is no, you should generally avoid moving to 24 
i4i.4xlarge
nodes for a sustained production workload like StarRocks. While the "up to 25 Gbps" figure looks attractive, it is a burstable limit (credit-based), not a baseline. Here is a breakdown of why your query performance dropped and why the 
4xlarge
strategy will likely bring back your bandwidth issues. 1. The "Up to" Bandwidth Trap On AWS, instance sizes 
4xlarge
and smaller use a "bucket" credit system for networking. * `i4i.4xlarge`: Has a baseline of only 5 Gbps. It can burst to 25 Gbps for short periods (usually 5–60 minutes), but once your credits are gone, you will be throttled to 5 Gbps. * `i4i.8xlarge`: Has a baseline (sustained) bandwidth of 18.75 Gbps. It is not "up to"; it is a guaranteed floor. * `i4i.16xlarge`: Has a baseline of 37.5 Gbps. Result of your proposal: Moving to 24 nodes of 
4xlarge
would give you a total cluster baseline of $24 \times 5 = 120 \text{ Gbps}$. Your original 12-node 
8xlarge
cluster had $12 \times 18.75 = 225 \text{ Gbps}$ of baseline bandwidth. You would be effectively cutting your cluster's sustained network capacity in half. 2. Why Query Performance Dropped on 6 Nodes When you consolidated from 12 nodes to 6 nodes, you reduced the Massive Parallelism of the cluster. StarRocks is an MPP (Massively Parallel Processing) engine. * Scan Parallelism: With 12 nodes, you had 12 different machines reading data from S3 and local cache simultaneously. With 6 nodes, even if they are larger, you have fewer "entry points" for the initial scan. * Cache Contention: Each of your 6 nodes is now responsible for managing twice as much cached data. This increases internal lock contention and can lead to "hot" CPU cores while others are idle. * Shuffling: In complex joins, data is "shuffled" between nodes. With fewer nodes, more data is forced through fewer physical network interfaces, which can lead to local congestion even if the "total" cluster bandwidth is theoretically the same. 3. How to fix the 6-node performance Since you already have the 6 
16xlarge
nodes and the bandwidth issues are gone, you can likely recover the performance by tuning StarRocks to utilize the larger number of cores per node: * Increase `parallel_fragment_exec_instance_num`: This session variable (or global config) controls how many instances of a query fragment run on a single node. If this was set for your 32-core 
8xlarge
nodes (e.g., at 16 or 32), it is likely too low for your 64-core 
16xlarge
nodes. Try setting this to 32 or 48. * Adjust `pipeline_dop`: Ensure your pipeline engine Degree of Parallelism is set to 0 (auto) or manually increased to match the higher core count. * Check Bucket Count: If your tables were bucketed for 12 nodes (e.g., 24 or 48 buckets), they may now be "imbalanced" on 6 nodes. Ensure your total bucket count is a multiple of your node count (6). 4. Better Topology Recommendation If you need more bandwidth than the original 12 
8xlarge
nodes provided, but want to keep the parallelism of a 12-node cluster, consider 
i4i.12xlarge
. * 
i4i.12xlarge
 has a baseline of 28.12 Gbps. * 12 nodes of 
12xlarge
would give you 337 Gbps of total cluster baseline (a 50% increase over your original setup) while maintaining your 12-way scan parallelism. Summary: Stick with the 
16xlarge
nodes and tune your parallelism variables, or scale out to 12 nodes of 
12xlarge
. *Do not move to `4xlarge`*; the 5 Gbps baseline will eventually throttle your ingestion and compaction into the ground once the burst credits are exhausted.
m
@Rocky
Copy code
MySQL [requests]> explain analyze select max(received) from requests;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String                                                                                                                                            |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Summary                                                                                                                                           |
|     QueryId: 019efc0f-2de6-70bc-9a70-b2e206ec7aaa                                                                                                 |
|     Version: 4.0.7-b75f536                                                                                                                        |
|     State: Finished                                                                                                                               |
|     TotalTime: 9s870ms                                                                                                                            |
|         ExecutionTime: 9s848ms [Scan: 9s613ms (97.61%), Network: 1.260ms (0.01%), ResultDeliverTime: 0ns (0.00%), ScheduleTime: 9s847ms (99.99%)] |
|         CollectProfileTime: 5ms                                                                                                                   |
|         FrontendProfileMergeTime: 1.401ms                                                                                                         |
|     QueryPeakMemoryUsage: ?, QueryAllocatedMemoryUsage: 1.939 TB                                                                                  |
|     Top Most Time-consuming Nodes:                                                                                                                |
|         1. OLAP_SCAN (id=0) : 9s664ms (99.70%)                                                                                               |
|         2. TOP_N (id=1) [ROW_NUMBER, TOP-N]: 27.299ms (0.28%)                                                                                     |
|         3. MERGE_EXCHANGE (id=2) [GATHER]: 1.473ms (0.02%)                                                                                        |
|         4. RESULT_SINK: 71.948us (0.00%)                                                                                                          |
|         5. AGGREGATION (id=3) [finalize, update]: 65.397us (0.00%)                                                                                |
|     Top Most Memory-consuming Nodes:                                                                                                              |
|     NonDefaultVariables:                                                                                                                          |
|         enable_adaptive_sink_dop: false -> true                                                                                                   |
|         enable_async_profile: true -> false                                                                                                       |
|         enable_profile: false -> true                                                                                                             |
| Fragment 0                                                                                                                                        |
| │   BackendNum: 1                                                                                                                                 |
| │   InstancePeakMemoryUsage: 231.797 KB, InstanceAllocatedMemoryUsage: 250.898 KB                                                                 |
| │   PrepareTime: ?                                                                                                                                |
| └──RESULT_SINK                                                                                                                                    |
|    │   TotalTime: 71.948us (0.00%) [CPUTime: 71.948us]                                                                                            |
|    │   OutputRows: 1                                                                                                                              |
|    │   SinkType: MYSQL_PROTOCAL                                                                                                                   |
|    └──AGGREGATION (id=3) [finalize, update]                                                                                                       |
|       │   Estimates: [row: 1, cpu: 8.00, memory: 8.00, network: 0.00, cost: 6361690388.00]                                                        |
|       │   TotalTime: 65.397us (0.00%) [CPUTime: 65.397us]                                                                                         |
|       │   OutputRows: 1                                                                                                                           |
|       │   PeakMemory: ?, AllocatedMemory: ?                                                                                                       |
|       │   AggExprs: [max(1: received)]                                                                                                            |
|       └──MERGE_EXCHANGE (id=2) [GATHER]                                                                                                           |
|              Estimates: [row: 1, cpu: 8.00, memory: 8.00, network: 8.00, cost: 6361690368.00]                                                     |
|              TotalTime: 1.473ms (0.02%) [CPUTime: 212.990us, NetworkTime: 1.260ms]                                                                |
|              OutputRows: 1                                                                                                                        |
|              PeakMemory: ?, AllocatedMemory: ?                                                                                                    |
|                                                                                                                                                       |
| Fragment 1                                                                                                                                        |
| │   BackendNum: 6                                                                                                                                 |
| │   InstancePeakMemoryUsage: 3.090 GB, InstanceAllocatedMemoryUsage: 1.939 TB                                                                     |
| │   PrepareTime: ?                                                                                                                                |
| └──DATA_STREAM_SINK (id=2)                                                                                                                        |
|    │   PartitionType: UNPARTITIONED                                                                                                               |
|    └──TOP_N (id=1) [ROW_NUMBER, TOP-N]                                                                                                            |
|       │   Estimates: [row: 1, cpu: 2544676128.00, memory: 8.00, network: 2544676128.00, cost: 6361690336.00]                                      |
|       │   TotalTime: 27.299ms (0.28%) [CPUTime: 27.299ms]                                                                                         |
|       │   OutputRows: 6                                                                                                                           |
|       │   PeakMemory: ?, AllocatedMemory: ?                                                                                                       |
|       │   OrderByExprs: [<slot 1> 1: received]                                                                                                    |
|       └──OLAP_SCAN (id=0)                                                                                                                    |
|              Estimates: [row: 318084516, cpu: 2544676128.00, memory: 0.00, network: 0.00, cost: 1272338064.00]                               |
|              TotalTime: 9s664ms (99.70%) [CPUTime: 51.424ms, ScanTime: 9s613ms]                                                              |
|              OutputRows: 330.211M (330210706)                                                                                                |
|              RuntimeFilter: 330.211M (330210706) -> 330.211M (330210706) (0.00%)                                                             |
|              Table: : requests                                                                                                               |
|              Detail Timers: [ScanTime = IOTaskExecTime + IOTaskWaitTime]                                                                     |
|                  IOTaskExecTime: 791.954ms [min=327.285us, max=9s612ms]                                                                      |
|                      CreateSegmentIter: 578.875ms [min=34ns, max=7s437ms]                                                                    |
|                  IOTaskWaitTime: 422.917us [min=5.526us, max=13.013ms]                                                                       |
|                                                                                                                                                       |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
THis simple query took almost 10 seconds. What is the issue here.
r
Based on the 
explain analyze
output, your query is suffering from a massive metadata and I/O initialization bottleneck, compounded by the fact that the engine is performing a full column scan of 330 million rows rather than an optimized lookup. Here are the specific issues identified in your profile: 1. The "Smoking Gun": 
CreateSegmentIter
(7.4s) Look at the 
Detail Timers
for the 
OLAP_SCAN
node: * `CreateSegmentIter`: 578.875ms [min=34ns, max=7s437ms] This is the most critical metric. In a StarRocks shared-data (Lake) cluster, 
CreateSegmentIter
is the time spent opening the segment files and reading their metadata (footers/headers) from S3. * The Issue: One (or more) of your nodes took 7.4 seconds just to open the files before it even started reading the actual data. * Why this happens: This is almost always caused by S3 Request Throttling or high Network Latency when communicating with the S3 API. Even if your "Bandwidth Allowance" is no longer exceeded, you might be hitting S3 Prefix limits (3,500 PUT/5,500 GET requests per second per prefix) if your data is not partitioned well. 2. Lack of Aggregate Pushdown (
OutputRows: 330.211M
) The plan shows that the 
OLAP_SCAN
node is reading and outputting all 330 million rows to the next operator (
TOP_N
). * The Issue: Ideally, for a 
max()
query, the scan node should only output the local maximum for each tablet (a few hundred rows total). Instead, it is streaming every single value of the 
received
column through the execution pipeline. * Why this happens: This usually occurs if the 
received
column is not part of the Sort Key (the 
ORDER BY
clause in your 
CREATE TABLE
statement). If it isn't in the sort key, StarRocks cannot use the "Zone Map" (min/max indexes) to skip data and must read the entire column. 3. Optimizer Strategy: 
TOP_N
vs 
AGGREGATE
 The optimizer chose a 
TOP_N
strategy (sorting the column to find the 1st row) instead of a standard 
AGGREGATE
(find max). * While this is often an optimization for sort-key columns, here it is clearly failing because it's still forcing a full scan of 330M rows across the network. 4. Abnormal Memory Usage (
1.939 TB
) The profile reports 
QueryAllocatedMemoryUsage: 1.939 TB
. For a query on 330 million rows (which is roughly 2.5 GB of raw data for a timestamp), this is extremely abnormal. * This suggests the system is struggling to manage the file handles or memory buffers during that long 7-second "Wait" period, or there is significant overhead in the metadata layer. ────────── How to fix this: 1. Check your 
ORDER BY
clause: If 
received
is a column you frequently run 
MAX()
or 
MIN()
on, or use in 
WHERE
filters, it must be included in the 
ORDER BY
(Sort Key) in your table definition. _ _If it is already in the sort key:* Run 
ANALYZE TABLE requests;
to update statistics, as the optimizer might be making a poor choice based on stale data. 2. Verify Data Caching: Run the query twice. If the second run is significantly faster (e.g., < 500ms), then the 9-second delay was caused by "Cold" data being fetched from S3. If the second run is still slow, the issue is likely the metadata overhead or S3 API latency. 3. Check S3 Latency Metrics: Since you are on a Lake (shared-data) cluster, check your AWS S3 CloudWatch metrics for 
TotalRequestLatency
and 
5xxErrors
. If you see spikes during your query, you need to re-shard your data or use more S3 prefixes. 4. Try forcing a different plan: See if a standard aggregation performs better by disabling the Top-N optimization for this session:
Copy code
sql
    SET enable_sort_aggregate = false;
    SELECT max(received) FROM requests;
Summary: Your nodes are "stalling" for 7 seconds while trying to talk to S3 to open files. This is either an S3 bottleneck or a sign that your 
received
column needs to be part of the table's Sort Key to avoid a 330-million-row scan.
m
@Rocky
Copy code
PRIMARY KEY(`received`, `auction_id`, `imp_key`)
COMMENT "OLAP"
PARTITION BY date_trunc('hour', received)
DISTRIBUTED BY HASH(`auction_id`) BUCKETS 128
ORDER BY(`received`, `auction_id`)
PROPERTIES (
"compression" = "ZSTD",
"datacache.enable" = "true",
"datacache.partition_duration" = "25 hours",
"enable_async_write_back" = "false",
"enable_persistent_index" = "true",
"file_bundling" = "true",
"partition_live_number" = "72",
"persistent_index_type" = "CLOUD_NATIVE",
"replication_num" = "1",
"storage_volume" = "core_services_volume"
); |
THis is my table properties. I should be having 100% cache hit rate (NVMe). Why is it hitting S3.
r
The 10-second latency for a simple 
max()
query on a 330-million-row table is caused by the specific architectural overhead of Primary Key (PK) tables in Shared-Data (Lake) mode. Even though you have NVMe caching and the column is in the sort key, the query is performing a full scan and stalling on S3 metadata. Here is why: 1. Why it's scanning 330M rows (PK Table Constraint) In a Duplicate or Aggregate table, 
max(received)
would take milliseconds because the engine only needs to check the "Zone Map" (min/max indexes) in the segment footers. However, in a Primary Key table: * The Delete Vector Problem: The engine cannot trust the segment's min/max index alone. Because PK tables support updates and deletes, the "maximum" value stored in a segment might belong to a row that has since been deleted or updated in a newer version. * Full Scan Requirement: To guarantee the correct result, StarRocks must read the column and apply the Delete Vector (DelVector) to every row to ensure the row is still "alive." This forces the 
OLAP_SCAN
node to process all 330 million rows rather than just looking at metadata. 2. Why it's hitting S3 despite 100% Cache Hit Rate The 
Explain Analyze
shows the bottleneck is 
CreateSegmentIter
(7.4 seconds). This is not reading "data" blocks (which are likely in your NVMe cache); it is the time spent opening and initializing segments. * Metadata is on S3: In shared-data mode, segment footers, PK indexes, and Delete Vectors are stored on S3. While StarRocks tries to cache these, they are often the first things evicted or may require frequent re-validation against the Frontend (FE). * The "Many Files" Penalty: If your table has many small files (check your compaction score with 
SHOW PROC '/dbs/<db_id>/<table_id>/partitions'
), the engine must make an S3 call to fetch the metadata for every small file to initialize the scan. 7.4 seconds is a classic symptom of S3 API Latency or Request Throttling when opening hundreds/thousands of files. * Persistent Index (
CLOUD_NATIVE
): Your index is stored on S3. To process the scan and apply the DelVectors, the engine must ensure the relevant parts of the PK index are available. 3. The 1.9 TB Memory Usage The 
InstanceAllocatedMemoryUsage: 1.939 TB
is extremely high for this query. * In PK tables, the Primary Key Index must be loaded into memory to track row locations. * For 330M rows, the raw index should be around 20–40 GB. Seeing 1.9 TB suggests the query is tracking a massive amount of metadata or that your segments are so fragmented that the overhead of managing them is consuming all available memory. ────────── Recommended Actions 1. Check Compaction Score: Run 
SHOW PROC '/dbs/<your_db_id>/<your_table_id>/partitions';
and look at the 
MaxCS
(Max Compaction Score) column. If this value is high (e.g., > 100), your 
CreateSegmentIter
is slow because it's opening too many small files. _ _Fix:* Manually trigger a compaction or reduce the frequency of small loads. 2. Verify Cache Hit Metrics: Run the query and then check the full profile for 
DataCacheReadBytes
vs 
BytesRead
. If 
DataCacheReadBytes
is significantly lower than 
BytesRead
, your metadata (footers) is not being cached, likely due to the 
datacache.partition_duration
(25 hours) being too short for the data you are querying. 3. Optimize the Table Type (If Applicable): If your primary use case for this table is tracking the 
max(received)
(e.g., for a watermark) and you don't actually need to update old rows frequently, an Aggregate Table or a Duplicate Table with a properly tuned sort key will be 100x faster for this specific query. 4. Warm up the Metadata: If this is a critical query, you can try 
CACHE SELECT max(received) FROM requests;
to force the metadata into the local cache, though this is a temporary fix.
Summary: The 10s delay is "Metadata Stall." The engine is waiting on S3 to open thousands of segment files so it can apply Delete Vectors to 330M rows, which is mandatory for Primary Key tables. References • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md • faq/loading/Loading_faq.md
m
@Rocky How do I check the whether the query reads from cache
r
To check whether your query is hitting the NVMe cache (Data Cache) in a StarRocks Lake (Shared-data) cluster, you need to examine the detailed Query Profile. While your 
EXPLAIN ANALYZE
summary shows high-level timings, it doesn't always include the low-level cache counters. You can find these by looking at the Full Profile (available via 
show profile for query '<query_id>';
or in the StarRocks Manager UI). 1. Key Metrics to look for: Under the 
OLAP_SCAN
node (or 
HdfsScanNode
in some versions), look for these three metrics: * 
DataCacheReadBytes
: The amount of data read directly from your NVMe/SSD cache. * 
BytesRead
: The total amount of data the query requested. * 
DataCacheWriteBytes
: The amount of data that was fetched from S3 and then written to your local cache because it wasn't there already. Interpret the results: * 100% Cache Hit: 
DataCacheReadBytes
== 
BytesRead
(and 
DataCacheWriteBytes
is 0). * Cache Miss (Cold Read): 
DataCacheWriteBytes
is high, and 
DataCacheReadBytes
is low/zero. * Partial Hit: 
DataCacheReadBytes
is significantly lower than 
BytesRead
. 2. Why your 9s query might still show a "100% Hit" In your specific case, you saw a 7.4s delay in 
CreateSegmentIter
. This timing represents Metadata initialization, not data scanning. The Data Cache primarily caches data blocks. The segment metadata (footers/headers) and Primary Key indexes are managed differently: * Segment/Footer Cache: These are small pieces of metadata at the end of files. Even if your "Data Cache" hit rate is 100%, if you have thousands of files, the engine may still be making thousands of small S3 "HeadObject" or "GetRange" requests to validate that the files haven't changed. * Primary Key Index: In 
CLOUD_NATIVE
mode, the index itself lives on S3. If it's not in the local persistent index cache, it must be fetched before the scan can even begin. 3. How to verify if the Cache is even "Attempted" If you don't see the 
DataCache
metrics in your profile at all, the query might not be using the cache. You can verify the cache plan by running:
Copy code
sql
EXPLAIN VERBOSE SELECT max(received) FROM requests;
Look for 
dataCacheOptions={populate: true}
in the scan node. If it says 
false
, you can force it for your session:
Copy code
sql
SET populate_datacache_mode = 'always';
Recommendation If your 
DataCacheReadBytes
matches your 
BytesRead
but the query is still slow: 1. Check file count: Run 
show tablets from requests
to see how many tablets/files are being opened. 2. Compaction: If you have too many small files, the metadata overhead (S3 latency) will dwarf the actual data reading time. Performace, regardless of your NVMe speed. References • data_source/data_cache.md • data_source/data_cache_troubleshooting.md • using_starrocks/caching/query_cache.md • data_source/data_cache_observe.md
m
@Rocky MySQL [requests]> SHOW TABLET FROM requests PARTITION(p2026062514) -> ; +----------+-----------+----------+----------+------------+ | TabletId | BackendId | DataSize | RowCount | MinVersion | +----------+-----------+----------+----------+------------+ | 269933 | [215620] | 3.2GB | 1990329 | 137 | | 269934 | [215620] | 3GB | 1988098 | 137 | | 269935 | [215620] | 3.1GB | 1988711 | 137 | | 269936 | [215607] | 3.2GB | 1989207 | 137 | | 269937 | [215602] | 3.2GB | 1991096 | 137 | | 269938 | [215620] | 3.2GB | 1989998 | 137 | | 269939 | [232880] | 3.2GB | 1991617 | 137 | | 269940 | [215623] | 3.2GB | 1988282 | 137 | | 269941 | [215620] | 3.2GB | 1989454 | 137 | | 269942 | [215623] | 3.2GB | 1989234 | 137 | | 269943 | [232880] | 3.2GB | 1989302 | 137 | | 269944 | [215607] | 3GB | 1989834 | 137 | | 269945 | [232880] | 3.2GB | 1988948 | 137 | | 269946 | [215602] | 3.1GB | 1989369 | 137 | | 269947 | [215613] | 3.2GB | 1989687 | 137 | | 269948 | [215620] | 3.2GB | 1988869 | 137 | | 269949 | [215602] | 3.2GB | 1987547 | 137 | | 269950 | [232880] | 3.1GB | 1988646 | 137 | | 269951 | [215607] | 3.2GB | 1989155 | 137 | | 269952 | [232880] | 3.2GB | 1987183 | 137 | | 269953 | [215620] | 3.2GB | 1989050 | 137 | | 269954 | [215607] | 3.2GB | 1987706 | 137 | | 269955 | [232880] | 3.2GB | 1989130 | 137 | | 269956 | [215602] | 3.2GB | 1990155 | 137 | | 269957 | [232880] | 3.2GB | 1989810 | 137 | | 269958 | [215607] | 3.2GB | 1987551 | 137 | | 269959 | [232880] | 3.2GB | 1990217 | 137 | | 269960 | [215613] | 3.2GB | 1987997 | 137 | | 269961 | [215602] | 3.1GB | 1990949 | 137 | | 269962 | [215620] | 3.2GB | 1989212 | 137 | | 269963 | [215623] | 3.2GB | 1986988 | 137 | | 269964 | [215613] | 3.2GB | 1992229 | 137 | | 269965 | [232880] | 3.2GB | 1988003 | 137 | | 269966 | [215607] | 3.2GB | 1989303 | 137 | | 269967 | [215602] | 3.2GB | 1987726 | 137 | | 269968 | [215602] | 3.2GB | 1991771 | 137 | | 269969 | [215602] | 3.2GB | 1988845 | 137 | | 269970 | [215613] | 3GB | 1989551 | 137 | | 269971 | [215602] | 3.1GB | 1989736 | 137 | | 269972 | [215613] | 3.2GB | 1990026 | 137 | | 269973 | [215607] | 3.1GB | 1991332 | 137 | | 269974 | [215607] | 3GB | 1988762 | 137 | | 269975 | [215607] | 3.2GB | 1986452 | 137 | | 269976 | [232880] | 3.2GB | 1987145 | 137 | | 269977 | [215620] | 3.1GB | 1987424 | 137 | | 269978 | [215602] | 3.2GB | 1990641 | 137 | | 269979 | [215623] | 3GB | 1992292 | 137 | | 269980 | [232880] | 3.2GB | 1988415 | 137 | | 269981 | [232880] | 3.2GB | 1991212 | 137 | | 269982 | [215620] | 3.2GB | 1989790 | 137 | | 269983 | [215623] | 3.2GB | 1990331 | 137 | | 269984 | [232880] | 3.2GB | 1989778 | 137 | | 269985 | [215607] | 3.2GB | 1988888 | 137 | | 269986 | [215613] | 3.2GB | 1988405 | 137 | | 269987 | [215613] | 3.2GB | 1990338 | 137 | | 269988 | [215623] | 3.2GB | 1991208 | 137 | | 269989 | [215620] | 3.2GB | 1990312 | 137 | | 269990 | [215607] | 3.1GB | 1988598 | 137 | | 269991 | [215623] | 3.2GB | 1988851 | 137 | | 269992 | [215623] | 3.2GB | 1989344 | 137 | | 269993 | [215620] | 3.2GB | 1991379 | 137 | | 269994 | [215602] | 3.2GB | 1988604 | 137 | | 269995 | [215602] | 3.2GB | 1986877 | 137 | | 269996 | [215620] | 3.2GB | 1989190 | 137 | | 269997 | [215623] | 3.1GB | 1991211 | 137 | | 269998 | [215623] | 3.2GB | 1986499 | 137 | | 269999 | [215602] | 3.1GB | 1986824 | 137 | | 270000 | [215623] | 3.2GB | 1990260 | 137 | | 270001 | [215613] | 3.1GB | 1987893 | 137 | | 270002 | [215623] | 3.2GB | 1991909 | 137 | | 270003 | [232880] | 3.2GB | 1991413 | 137 | | 270004 | [215620] | 3GB | 1988536 | 137 | | 270005 | [215613] | 3.2GB | 1989766 | 137 | | 270006 | [215623] | 3.2GB | 1988947 | 137 | | 270007 | [232880] | 3.2GB | 1991040 | 137 | | 270008 | [215607] | 3.1GB | 1990239 | 137 | | 270009 | [215623] | 3.2GB | 1987349 | 137 | | 270010 | [215607] | 3.2GB | 1987997 | 137 | | 270011 | [215613] | 3.2GB | 1991469 | 137 | | 270012 | [215620] | 3.2GB | 1989172 | 137 | | 270013 | [215607] | 3.2GB | 1989437 | 137 | | 270014 | [232880] | 3.2GB | 1988538 | 137 | | 270015 | [232880] | 3.2GB | 1989538 | 137 | | 270016 | [215607] | 3.2GB | 1991976 | 137 | | 270017 | [215613] | 3.2GB | 1986911 | 137 | | 270018 | [215602] | 3.2GB | 1988244 | 137 | | 270019 | [215620] | 3.2GB | 1991055 | 137 | | 270020 | [215607] | 3.2GB | 1989021 | 137 | | 270021 | [215623] | 3.2GB | 1990654 | 137 | | 270022 | [215620] | 3.1GB | 1990178 | 137 | | 270023 | [215623] | 3.2GB | 1991963 | 137 | | 270024 | [215620] | 3.1GB | 1990467 | 137 | | 270025 | [232880] | 3.1GB | 1987559 | 137 | | 270026 | [215613] | 3.1GB | 1993128 | 137 | | 270027 | [215613] | 3GB | 1985934 | 137 | | 270028 | [215602] | 3.1GB | 1990756 | 137 | | 270029 | [232880] | 3.2GB | 1986471 | 137 | | 270030 | [215607] | 3.2GB | 1988019 | 137 | | 270031 | [215607] | 3.2GB | 1988070 | 137 | | 270032 | [215613] | 3.2GB | 1989064 | 137 | | 270033 | [215602] | 3.2GB | 1990672 | 137 | | 270034 | [215623] | 3.2GB | 1990073 | 137 | | 270035 | [215623] | 3.2GB | 1988217 | 137 | | 270036 | [215620] | 3.1GB | 1989072 | 137 | | 270037 | [215613] | 3.2GB | 1990097 | 137 | | 270038 | [215623] | 3.2GB | 1988128 | 137 | | 270039 | [215602] | 3.2GB | 1987648 | 137 | | 270040 | [215623] | 3.2GB | 1988326 | 137 | | 270041 | [232880] | 3.1GB | 1989664 | 137 | | 270042 | [215623] | 3.2GB | 1989517 | 137 | | 270043 | [215602] | 3.2GB | 1992063 | 137 | | 270044 | [215607] | 3.2GB | 1989881 | 137 | | 270045 | [215613] | 3.2GB | 1987205 | 137 | | 270046 | [215602] | 3.2GB | 1989543 | 137 | | 270047 | [215602] | 3.1GB | 1990005 | 137 | | 270048 | [215613] | 3.2GB | 1991786 | 137 | | 270049 | [232880] | 3.2GB | 1988660 | 137 | | 270050 | [215607] | 3.1GB | 1989027 | 137 | | 270051 | [215623] | 3.2GB | 1990158 | 137 | | 270052 | [215620] | 3.1GB | 1989316 | 137 | | 270053 | [215607] | 3.2GB | 1990194 | 137 | | 270054 | [215613] | 3.2GB | 1988020 | 137 | | 270055 | [215613] | 3.2GB | 1987442 | 137 | | 270056 | [215620] | 3.1GB | 1990224 | 137 | | 270057 | [215602] | 3.2GB | 1990100 | 137 | | 270058 | [215620] | 3.2GB | 1989408 | 137 | | 270059 | [215613] | 3.1GB | 1988160 | 137 | | 270060 | [215613] | 3.2GB | 1988405 | 137 | +----------+-----------+----------+----------+------------+ These are my tablets on most recent partitions. Does tripling bucket count triple compaction parallelism? My cloud native compaction cant keep up with ingestion throughput (~120k rows/s)
r
Tripling your bucket count will not necessarily triple your compaction speed, and in your specific case (already seeing a 7.4s "Metadata Stall"), it will likely make your query performance worse. Here is the breakdown of why tripling buckets is risky and how to actually fix your compaction lag. 1. Does more buckets = more parallelism? Theoretically, yes. In StarRocks Shared-Data (Lake) mode, the tablet (bucket) is the unit of compaction. * The Benefit: If you have more tablets, the Frontend (FE) has more individual tasks it can schedule across your compute nodes. * The Limit: Compaction is ultimately throttled by two global settings: _ 
lake_compaction_max_tasks
(FE): Default is 
CN_nodes _ 16
. If you aren't hitting this cap, more buckets won't help. * 
compact_threads
(CN/BE): Default is 4. This is the most common bottleneck. Even if the FE sends 1,000 tasks, each node will only work on 4 at a time. 2. Why tripling buckets will hurt your 9s query Your previous 
explain analyze
showed that 7.4 seconds was spent in 
CreateSegmentIter
. This is the time it takes to "open" and fetch metadata for your files from S3. * Metadata Explosion: Tripling your bucket count from 128 to 384 means every query (and the compaction process itself) has to manage 3x as many file footers and segment metadata objects. * S3 Latency: You will triple the number of S3 
GET
requests required just to start a scan, likely increasing that 7.4s delay to over 20 seconds. * PK Table Overhead: In Primary Key tables, each tablet maintains its own Delete Vector and Index. More tablets = more memory fragmentation and more metadata to merge. 3. How to fix Compaction Lag for 120k rows/s For a high-throughput ingestion of 120k rows/s, you need to increase the intensity of compaction, not the number of shards. Step 1: Increase CN Parallelism Change the number of threads each node uses for compaction. The recommendation is 25% of your CPU cores.
Copy code
sql
-- Run this for each BE/CN node (or via global config)
UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";
Step 2: Increase FE Task Scheduling Ensure the FE is allowed to schedule enough concurrent tasks to keep the CNs busy.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "128"); -- Adjust based on cluster size
Step 3: Optimize PK Compaction Batching Increase the number of files merged in a single pass to reduce the total number of versions.
Copy code
sql
UPDATE information_schema.be_configs SET value = "500" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
Step 4: Verify the Lag Run this to see if tasks are actually running and if they are failing:
Copy code
sql
SELECT * FROM information_schema.be_cloud_native_compactions 
WHERE table_id = <your_table_id> 
ORDER BY start_time DESC;
If you see many "failed" tasks, check your CN logs; you might be hitting memory limits (OOM). Summary Recommendation Your current bucket count (128) and tablet size (~3GB) are actually very healthy. Do not change the bucket count. Instead, increase 
compact_threads
to 8 or 16 and ensure your ingestion batch interval is at least 5–10 seconds. High-frequency "micro-batches" (e.g., every 1 second) are the #1 cause of compaction lag in Primary Key tables. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/resource_management/query_queues.md
m
@Rocky I've set the configs you recommended but issue is my concurrent compaction tasks are nowhere near the ceiling.
r
If you have increased the global limits (
lake_compaction_max_tasks
and 
compact_threads
) but the actual concurrency remains low, you are likely hitting the per-partition parallelism limit. In StarRocks Lake (Shared-data) mode, the FE schedules compaction at the partition level. Even if you have 128 buckets, the FE throttles how many of those buckets are compacted simultaneously within a single partition to avoid overwhelming the cluster or the S3 commit queue. 1. The "Hidden" Bottleneck: 
lake_compaction_max_parallel
 The most common reason for low concurrency in a wide table (128 buckets) is the 
lake_compaction_max_parallel
setting. * Default: Typically 3. * What it does: It limits the number of parallel subtasks (tablets/buckets) that can be scheduled per partition. * Effect: Even if you have 128 buckets ready to be compacted in your active partition, the FE will only send 3 at a time to the CN nodes. How to fix it: You can increase this globally in the FE or specifically for your table. For a table with 128 buckets, you can safely increase this to 16 or 32 to utilize more CN resources:
Copy code
sql
-- Increase globally in FE
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_parallel_default" = "16");

-- OR specifically for the table (preferred)
ALTER TABLE requests SET ("lake_compaction_max_parallel" = "16");
2. High-Frequency Ingestion (The 10s Rule) You mentioned 120k rows/s. If you are doing very small, frequent commits (e.g., every 1-2 seconds), you are creating "Rowset Bloat." * Compaction Score Calculation: The score is based on the number of versions. If you commit 10 times a minute, you generate 600 versions per hour. * The Conflict: Compaction itself is a transaction. If a new ingestion commit happens while a compaction is "committing," they can block each other. * Fix: Ensure your ingestion batch interval (e.g., in Flink or Stream Load) is at least 10–15 seconds. Larger batches = fewer versions = lower compaction score. 3. Check for "Stuck" Tasks If the parallelism is low because tasks are failing or hanging, the FE won't schedule new ones. Check the status of current/recent tasks:
Copy code
sql
SELECT * FROM information_schema.be_cloud_native_compactions 
WHERE table_id = (SELECT id FROM information_schema.tables WHERE table_name = 'requests')
ORDER BY start_time DESC LIMIT 20;
* Look for `status`: If you see 
ABORTED
or many tasks with a long 
duration
, check the 
CN
logs for OOM or S3 timeout errors. * 
CommitTime
vs `FinishTime`: If 
CommitTime
is filled but 
FinishTime
is null, the task is waiting on the FE to publish the version metadata. 4. Optimize PK-Specific Compaction For Primary Key tables, compaction is more expensive because it has to merge the Primary Key Index. Ensure these BE-side configs are tuned:
Copy code
sql
-- Allow more rowsets to be merged in one go for PK tables
UPDATE information_schema.be_configs SET value = "1000" WHERE name = "lake_pk_compaction_max_input_rowsets";

-- Accelerate the S3 I/O during compaction
UPDATE information_schema.be_configs SET value = "4194304" WHERE name = "lake_compaction_stream_buffer_size_bytes"; -- 4MB
Summary Recommendation 1. Increase 
lake_compaction_max_parallel
 to 16 or higher. 2. Check 
information_schema.be_cloud_native_compactions
to ensure tasks aren't failing. 3. Ensure your ingestion batch size is large enough so you aren't committing more than once every 10 seconds.
m
@Rocky If my data is uniformly distributed by my distribution key, I shoudlt be concerned by a hot tablet right? Because data is distributed across tablets evenly, each tablet can have compaction running on it? if that's the case, I have 128 tablets per partition across 6 machines for compaction to run on, why am I stuck on average of 1-2 concurrent compactions
r
Your logic about uniform distribution is correct, but you are hitting a fundamental design constraint of how StarRocks Lake (Shared-data) mode handles compaction transactions. Even though you have 128 tablets and 6 machines, the reason you are stuck at ~2 concurrent tasks is likely due to the "One Partition, One Transaction" rule and the Subtask Parallelism limit. 1. The Bottleneck: One Compaction Transaction per Partition In StarRocks Lake mode, the Frontend (FE) schedules compaction by partition. * For any given partition (e.g., 
p2026062514
), the FE initiates exactly one compaction transaction at a time. * It does not matter if you have 128 tablets; if they all belong to the same active partition, they are all part of a single transaction "job." * If you only have one partition receiving your 120k rows/s, you effectively only have one job running. 2. The Solution: 
lake_compaction_max_parallel
 Within that single partition-level job, the FE limits how many tablets (subtasks) are sent to the CN nodes simultaneously. * The Default: The FE parameter 
lake_compaction_max_parallel_default
is set to 3. * The Result: Even if your partition has 128 tablets ready to compact, the FE will only dispatch 3 of them at a time to your 6 nodes. This explains why you see only 1–3 concurrent tasks in your metrics. How to fix it: You need to tell the FE to allow more tablets within that single partition transaction to run in parallel. For 128 tablets and 6 nodes, a value of 16 or 32 is appropriate:
Copy code
sql
-- Increase the global default for all future tables/partitions
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_parallel_default" = "16");

-- Force it onto your existing table immediately
ALTER TABLE requests SET ("lake_compaction_max_parallel" = "16");
3. Check the "Compaction Score" Even with uniform distribution, a tablet won't compact until its "Score" (number of versions) hits a certain threshold. * Check scores: Run 
SHOW PROC '/compactions';
* Look for your partition. If the 
MaxScore
is just hovering around 10 (the default 
lake_compaction_score_selector_min_score
), the FE won't schedule more tasks because it thinks the tablets are "clean enough." * If you want compaction to be more aggressive (merging smaller files sooner), you can lower the threshold:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "5");
4. Why "Hot Tablets" aren't your problem You are right that uniform distribution prevents a "hot tablet" (where one tablet has a massive score and others are 0). However, in your case, you have the opposite problem: Uniformly Low Scores. Because data is spread across 128 tablets, it takes much longer for any single tablet to accumulate enough versions to trigger the compaction threshold compared to if you had only 10 tablets. Summary of adjustments: 1. Increase 
lake_compaction_max_parallel
to 16. This allows your 6 nodes to work on 16 tablets at once for your single active partition. 2. Verify 
compact_threads
is at least 8 or 16 on each CN node (which you've already done). 3. Check 
SHOW PROC '/compactions'
 to ensure the FE is actually seeing scores high enough to trigger tasks. References • administration/management/compaction.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md References • administration/management/compaction.md
m
@Rocky MySQL [(none)]> admin show frontend config like '%lake_compaction_max%'; +---------------------------+------------+-------+------+-----------+-----------------------------------------------------------------------------------------------+ | Key | AliasNames | Value | Type | IsMutable | Comment | +---------------------------+------------+-------+------+-----------+-----------------------------------------------------------------------------------------------+ | lake_compaction_max_tasks | [] | 96 | int | true | -1 means calculate the value in an adaptive way. set this value to 0 will disable compaction. | +---------------------------+------------+-------+------+-----------+-----------------------------------------------------------------------------------------------+ 1 row in set (0.001 sec) MySQL [(none)]> Why is lake_compaction_max_parallel_default not available to me. I'm on version 4.0.7-b75f536
r
It appears you have hit a version-specific limitation. The configuration 
lake_compaction_max_parallel_default
was introduced in StarRocks v4.1 as part of a major enhancement to parallelize cloud-native compaction within a single partition. In your version (v4.0.7), StarRocks uses an earlier scheduling logic where compaction is strictly partition-oriented. Even with 128 tablets and 6 nodes, the FE typically initiates only one or two compaction transactions per active partition at a time. This explains why your concurrency is stuck at 1–2 despite your high ceilings. To resolve this in version 4.0, you should focus on reducing the "small file storm" and maximizing the efficiency of the tasks that do run. 1. Enable File Bundling (The v4.0 Solution) Version 4.0 introduced File Bundling specifically to solve the "too many small files" issue in shared-data clusters at the ingestion layer. This significantly reduces the pressure on compaction by merging small rowsets before they are written to S3. Run this on your BE/CN nodes:
Copy code
sql
UPDATE information_schema.be_configs SET value = "true" WHERE name = "lake_enable_file_bundling";
2. Optimize Primary Key Compaction If you are using Primary Key tables, ensure the Size-Tiered strategy is active. This strategy allows StarRocks to be more selective and efficient with merges:
Copy code
sql
-- Ensure this is true (default in newer versions, but check)
UPDATE information_schema.be_configs SET value = "true" WHERE name = "enable_pk_size_tiered_compaction_strategy";

-- Allow each task to merge more rowsets at once to clear the backlog faster
UPDATE information_schema.be_configs SET value = "1000" WHERE name = "lake_pk_compaction_max_input_rowsets";
3. Trigger More "Jobs" via Partitioning Since v4.0 schedules by partition, if all your 120k rows/s are landing in one partition, you only get one "compaction slot." * The Workaround: If your business logic allows, use a finer partitioning scheme (e.g., partitioning by 
hour
instead of 
day
, or adding a "shard" column to your partition key). * Why it works: If 5 partitions are active and have high compaction scores, the FE will schedule 5 separate transactions, allowing you to hit your 
lake_compaction_max_tasks
limit and utilize all 6 machines. 4. Adjust the Compaction Threshold If you want the background thread to be more aggressive in picking up tasks:
Copy code
sql
-- Lower the score required to trigger a task (Default is 10)
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "5");
Summary for v4.0.7 You cannot use 
lake_compaction_max_parallel_default
to force tablet-level parallelism within one partition job. To get higher concurrency, you either need to upgrade to v4.1 or spread your ingestion across more partitions so the FE can schedule multiple partition-level jobs.
m
@Rocky The compaction situation wasn't nearly as bad, I could get queries very quickly when I had 12 i4i.8xlarge. I switched cluster topology to 6 i4i.16xlarge because I was hitting bandwidth exceeded allowance issue. How do I resolve this
r
The switch from 12 nodes to 6 nodes effectively halved your cluster's compaction "slots" and likely increased the network pressure per node, leading to the bottleneck you are seeing. Here is why the situation worsened and how to resolve it in StarRocks v4.0.7. 1. The "Transaction Limit" vs. Node Count In StarRocks Lake (Shared-data) mode, the FE limits the total number of concurrent compaction transactions based on the number of nodes. * The Math: By default, 
lake_compaction_max_tasks
is 
-1
(adaptive), which calculates to *`num_nodes * 16`*. * The Shift: * With 12 nodes, your limit was 192 concurrent compaction jobs. * With 6 nodes, your limit is now 96. * The Reality: Even though 96 is a high ceiling, in version 4.0.x, compaction is partition-oriented. The FE schedules exactly one compaction transaction per partition. If all your 120k rows/s are landing in just 1 or 2 active partitions, you will only see 1–2 rows in 
SHOW PROC '/compactions'
, regardless of how many tablets or nodes you have. 2. Why 12 nodes felt "Faster" Even if you only had one partition, 12 nodes were likely better for your bandwidth exceeded issue. * Network Ceiling: AWS 
i4i.8xlarge
nodes have a 10 Gbps baseline. With 12 nodes, you had an aggregate baseline of 120 Gbps. * The i4i.16xlarge Trap: While the 
16xlarge
has a higher 20 Gbps baseline, you now only have 120 Gbps aggregate. * Squeezing the I/O: Compaction in Lake mode reads data from S3, processes it, and writes it back to S3. With 6 nodes, each node is now responsible for twice as many tablets (21–22 tablets per node vs. 10–11). This forces each node to push double the network traffic to S3. If your ingestion + query traffic is already high, the compaction traffic is likely pushing the nodes past their 20 Gbps 
bw_out_allowance_exceeded
limit. 3. How to Resolve in v4.0.7 Since your version does not support the v4.1 
lake_compaction_max_parallel_default
(which allows many tablets to compact in parallel within one partition), you must use these "v4.0 tactics": A. Increase Partition Parallelism (The "Many Jobs" Fix) Since the FE can't parallelize inside one partition well in v4.0, you must parallelize by creating more partitions. * If you currently partition by 
day
, consider partitioning by 
hour
. * If 10 partitions are active and have versions to merge, the FE will launch 10 transactions. This will immediately utilize all 6 of your nodes and get you closer to your 96-task ceiling. B. Lower the Selection Threshold Force the FE to be more aggressive in picking up partitions that have even moderate amounts of new data:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "5");
C. Enable File Bundling (Highly Recommended) This is a game-changer for high-frequency ingestion in Lake mode. It merges small files in memory before writing them to S3, which reduces the "Rowset Bloat" that compaction has to clean up.
Copy code
sql
-- Run on all CN nodes
UPDATE information_schema.be_configs SET value = "true" WHERE name = "lake_enable_file_bundling";
D. Primary Key Specifics If you are on a PK table, ensure you are using the Size-Tiered strategy, which handles the "bandwidth exceeded" issue better by performing smaller, more frequent merges instead of massive, "stop-the-world" base merges.
Copy code
sql
UPDATE information_schema.be_configs SET value = "true" WHERE name = "enable_pk_size_tiered_compaction_strategy";
-- Accelerate the I/O for these tasks
UPDATE information_schema.be_configs SET value = "4194304" WHERE name = "lake_compaction_stream_buffer_size_bytes";
Summary Recommendation
Your switch to 6 larger nodes was a good move for node-level bandwidth limits, but it penalized your cluster-level compaction scheduling. To fix the 1-2 concurrent task limit, you must partition your data more finely (e.g., hourly) so the FE has more "jobs" to schedule simultaneously available" to schedule across your 6 beefy nodes. References • administration/management/FE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md
m
@Rocky Is there a way I can solve my compaction issues by making s3 api calls less infrequent? Like store everything on cache and perform all compaction or merges on cache first, then do s3 compaction or file uploads etc every configured interval like every 5 minutes
r
To achieve the "buffer locally and sync infrequently" behavior you are looking for in StarRocks v4.0.7 (Lake mode), you cannot technically "pause" S3 uploads for 5 minutes because StarRocks prioritizes durability. In a shared-data cluster, a transaction is only "committed" once the data is safely in S3. However, you can achieve nearly the same outcome (drastically reduced S3 API calls and bandwidth spikes) by combining File Bundling and Client-side Batching. 1. Enable File Bundling (The S3 API Reducer) This is the closest feature to your request. Instead of creating a separate S3 object for every tablet/rowset during every load or compaction, File Bundling aggregates these into fewer, larger files. * Why it helps: It reduces the number of 
PutObject
and 
List
calls to S3, which are often the source of the "bandwidth exceeded" and latency issues you encountered. * How to enable:
Copy code
sql
    -- For the cluster:
    UPDATE information_schema.be_configs SET value = "true" WHERE name = "lake_enable_file_bundling";
    -- For your specific table:
    ALTER TABLE <table_name> SET ("file_bundling" = "true");
2. Client-side Batching (The "5-Minute" Solution) Since StarRocks will attempt to write to S3 as soon as you send data, the most effective way to "perform merges on cache/memory first" is to buffer at the ingestion layer (e.g., in Flink, Kafka, or your custom loader). * If you send 120k rows every 1 second, you trigger thousands of S3 API calls per minute. * If you buffer those rows for 60–300 seconds and send one giant batch, StarRocks writes much larger, more efficient files to S3. This reduces the number of "versions" created, which in turn reduces the number of compaction tasks needed. 3. Ensure Compaction Uses the Local Cache You can optimize the CN nodes to ensure that when compaction does run, it reads from the local NVMe disks (the cache) instead of downloading data from S3 again. * Check your table property: Ensure 
"datacache.enable" = "true"
is set (this is the default). * Compaction in StarRocks is designed to check the local Data Cache first. As long as your cache is large enough to hold your "active" ingestion window, compaction will effectively be "merging on cache." 4. Why You Are Still Stuck at 1–2 Tasks Even if you optimize S3 calls, you still face the v4.0 scheduling bottleneck: The FE only allows one compaction transaction per partition at a time. * The Fix: You must use Finer Partitioning. * If you currently have one partition for "today," StarRocks will only ever run 1 compaction job for that data. * If you partition by 
hour
, and you have 6 hours of data currently "active" with high version counts, the FE will launch 6 concurrent jobs, allowing you to finally utilize your 6 nodes and 96-task ceiling. Summary Checklist for v4.0.7: 1. Enable 
file_bundling
 to reduce S3 API pressure. 2. Increase your ingestion batch interval to at least 30–60 seconds to reduce the number of small files created. 3. Switch to hourly partitions to allow the FE to schedule more than one compaction job at a time across your cluster. 4. Confirm 
lake_compaction_score_selector_min_score
is set to 5 to make the FE more eager to start tasks before the backlog gets too large. References • administration/management/FE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • data_source/data_cache.md • administration/management/compaction.md References • data_source/data_cache.md
m
@Rocky Is 
file_bundling
 only available in version 4.1?
r
No, 
file_bundling
is available in your version (v4.0.7). It was officially introduced in StarRocks v4.0.0 specifically to address the high S3 API costs and "small file storm" issues you are experiencing. While the "tablet-level parallel compaction" (controlled by 
lake_compaction_max_parallel_default
) is indeed a v4.1+ feature, the File Bundling logic is fully supported in 4.0.x. How to use it in v4.0.7: 1. Check the Global FE setting: It should be enabled by default for all new tables created in v4.0, but you can verify the global toggle:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'enable_file_bundling';
If it is 
false
, enable it:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_file_bundling" = "true");
2. Enable it for your existing table: If your table was created before the v4.0 upgrade (or if it was explicitly disabled), you must enable it at the table level:
Copy code
sql
    ALTER TABLE <table_name> SET ("file_bundling" = "true");
3. Verify the BE-side setting: On the BE/CN nodes, there is a corresponding parameter that governs the execution of bundling:
Copy code
sql
    -- Check if it exists and is true
    UPDATE information_schema.be_configs SET value = "true" WHERE name = "lake_enable_file_bundling";
Why it solves your problem: Without File Bundling, every time StarRocks writes data (ingestion or compaction), it creates a separate physical object in S3 for every tablet. If you have 128 tablets and ingest frequently, you generate thousands of small files and metadata updates per minute. * File Bundling packs these small data segments into a single larger physical file (a "bundle"). * This significantly reduces the number of 
PutObject
calls and 
List
calls to S3, which directly lowers the "bandwidth allowance" pressure on your 6 nodes. Note: If you enable this for an existing table, it will only apply to new data versions and new compaction tasks. It will not immediately bundle your old files until-now unbundled old files until they are picked up for a compaction cycle of compaction. References • administration/management/FE_configuration.md • administration/management/compaction.md • release_notes/release-4.0.md • knowledge/admin_and_operation/shared_data_compaction.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
m
@Rocky But if I change to 24 i4i.4xlarge, I quadruple compaction capacity but the outgoing data should be spread across more nodes right? So i shouldnt encounter bandwidth exceeded issues? Even then, is exceeding bandwidth a big problem?
r
Moving to 24 
i4i.4xlarge
 nodes is a significant improvement for your specific bottleneck, but it comes with a "v4.0 logic" caveat. Here is the breakdown of why this helps and what remains the same. 1. Bandwidth: 24 Nodes is Much Better Your assumption is correct. Spreading the load across more nodes is the standard way to bypass AWS network limits (
bw_out_allowance_exceeded
). * Aggregate Capacity: 24 
i4i.4xlarge
nodes have an aggregate baseline of 240 Gbps (24 x 10 Gbps). This is double the 120 Gbps aggregate of your 6 
i4i.16xlarge
nodes. * Per-Node Pressure: With 24 nodes, each node only handles ~5-6 tablets. When compaction reads from and writes to S3, the "per-node" network traffic is cut by 75% compared to your 6-node setup. This makes it much harder to hit the 10 Gbps ceiling on any single node. 2. Compaction Capacity: The "384 Slot" Trap While your theoretical compaction capacity increases to 384 (24 nodes * 16), in StarRocks v4.0.7, you will still likely see only 1–2 rows in 
SHOW PROC '/compactions'
unless you change your partitioning. * The v4.0 Bottleneck: In your version, one "compaction task" equals one partition. * If you have 120k rows/s landing in only one partition, StarRocks will only ever run one compaction transaction for that data, regardless of whether you have 6 nodes or 24 nodes. * The Solution: You must use finer partitioning (e.g., partition by 
hour
instead of 
day
) to allow the FE to launch multiple transactions (e.g., 24 tasks for 24 hours of data) that can then be distributed across your 24 nodes. 3. Is "Exceeding Bandwidth" a Big Problem? Yes, it is critical. In StarRocks Lake (Shared-data) mode, hitting network limits causes a cascade of failures: 1. Ingestion Latency: Data cannot be written to S3 fast enough. Your ingestion backpressure will rise, and eventually, the 120k rows/s stream will lag. 2. Compaction Failure: Compaction requires reading data from S3, merging it, and writing it back. If the network is throttled, compaction takes longer than the ingestion interval. You will start seeing the "Too many versions" error, which eventually stops all ingestion to the table. 3. Query Slowdown: If the network is saturated by compaction/ingestion, queries that need to pull data from S3 will experience massive latency spikes. 4. Node Instability: Extreme network saturation can lead to failed heartbeats between the FE and CN, causing nodes to be marked as "Dead" temporarily, which further reduces your capacity and worsens the cycle. Summary Recommendation Switching to 24 smaller nodes is the right move for network stability and aggregate throughput. However, to actually use those 24 nodes for compaction in v4.0.7: 1. Partition more finely: Ensure you have multiple "active" partitions receiving data or needing compaction. 2. Enable `file_bundling`: (As discussed) to reduce the number of S3 API calls. 3. Set 
lake_compaction_score_selector_min_score
to 5: This forces the FE to start compaction tasks earlier, preventing the "Too many versions" cliff. If you don't want to change your partitioning logic, the only way to get more parallelism within a single partition is to upgrade to v4.1+, which introduces the 
lake_compaction_max_parallel
setting. References • administration/management/compaction.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md
m
@Rocky How to check concurrent compaction tasks being ran cluster wide?
r
To check concurrent compaction tasks being run cluster-wide in StarRocks Lake mode (v4.0.x), you can use the following commands: 1. The High-Level Overview (FE Level) Run this command to see the status of all compaction transactions scheduled by the FE:
Copy code
sql
SHOW PROC '/compactions';
How to interpret the results: * Active Tasks: Any row where the 
FinishTime
is NULL is currently running or being committed. * Scope: In v4.0.x, each row represents a Partition-level compaction task. * Columns: * `Partition`: The database, table, and partition name. * `TxnID`: Use this ID to drill down into specific tablet progress. * `Error`: If a task failed, the reason (e.g., S3 timeout, OOM) will appear here. ────────── 2. The Detailed Progress (Tablet/Node Level) To see which specific nodes (CNs) are working and how far along each tablet is, query the `information_schema`:
Copy code
sql
SELECT * FROM information_schema.be_cloud_native_compactions 
WHERE TXN_ID = <insert_txn_id_from_above>;
This is useful because a single "Partition compaction" from the 
SHOW PROC
command is actually split into many parallel tablet sub-tasks across your nodes. * PROGRESS: Shows the % completion for each tablet. * BE_ID: Shows which specific node is performing that specific tablet merge. ────────── 3. Check Current Capacity (Limits) If you want to know how many concurrent tasks your cluster could run vs. what it is actually running:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks';
* Value 
-1
(Default): This means your capacity is 
Number of Nodes * 16
. For your planned 24 nodes, this would be 384 concurrent tasks. * Value `0`: Compaction is disabled. Why you might see "low concurrency" despite high limits: In StarRocks v4.0.7, the FE only allows one compaction transaction per partition at a time. * If you have 24 nodes but only one massive partition receiving all your data, you will only see one row in 
SHOW PROC '/compactions'
. * To utilize all 24 nodes, you must have multiple active partitions (e.g., hourly partitions) so the FE can schedule multiple transactions concurrently. References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md • administration/management/FE_configuration.md References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md
m
@Rocky I mean prometheus metirc
r
To check the number of concurrent compaction tasks being run cluster-wide using Prometheus, you need to aggregate the metrics reported by each Compute Node (CN) or Backend (BE). Because StarRocks Lake mode (shared-data) uses an FE-driven scheduler to dispatch sub-tasks to nodes, the most accurate "cluster-wide" view of actual execution is the sum of active tasks across all nodes. 1. The Core Metrics You should sum the following metrics (all are Gauges): * 
starrocks_be_running_cumulative_compaction_task_num
: Active cumulative compaction tasks. * 
starrocks_be_running_base_compaction_task_num
: Active base compaction tasks. * 
starrocks_be_running_update_compaction_task_num
: Active Primary Key (PK) table compaction tasks. 2. PromQL for Cluster-Wide View To see the total number of compaction tasks currently running across your entire cluster, use this PromQL expression:
Copy code
promql
sum(
  starrocks_be_running_cumulative_compaction_task_num + 
  starrocks_be_running_base_compaction_task_num + 
  starrocks_be_running_update_compaction_task_num
)
3. Monitoring Compaction "Pressure" While the task count tells you how many nodes are busy, the Compaction Score tells you if you are falling behind. * Metric: 
starrocks_fe_tablet_max_compaction_score
* Meaning: This represents the highest compaction score in the cluster. If this value consistently exceeds 100, it indicates your compaction is not keeping up with ingestion. If it exceeds 1000, you will start seeing "Too many versions" errors. 4. Relevant Limits (For Reference) In v4.0.7, your cluster-wide capacity is governed by: * FE Level: 
lake_compaction_max_tasks
(Default: 
nodes * 16
). This is the maximum number of concurrent tasks the FE will dispatch cluster-wide. * CN Level: 
compact_threads
(Default: 
4
). This is the maximum number of threads per node dedicated to compaction. If your PromQL sum is significantly lower than your 
lake_compaction_max_tasks
limit despite having a high compaction score, it usually means you don't have enough active partitions to parallelize the work (since v4.0.x compaction is partition-level). References • administration/management/monitoring/metrics.md • administration/management/compaction.md • deployment/shared_data/feature-support-shared-data.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics-shared-data.md References • administration/management/monitoring/metrics-shared-data.md • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md
2 Views
Open in Slack
PreviousNext
Powered by