Sai
09/23/2026, 3:00 PMRocky
09/23/2026, 3:00 PMSai
09/23/2026, 3:02 PMRocky
09/23/2026, 3:02 PMSai
09/23/2026, 3:04 PMRocky
09/23/2026, 3:04 PMDELETE, UPDATE, and MERGE INTO directly on Iceberg tables.
* Iceberg V3: Full support for the Iceberg V3 specification and Variant data type.
* Paimon Support: Native INSERT support, along with Tag/Branch management and Time Travel for Paimon catalogs.
* Real-time & Scaling:
* Auto Tablet Split: Automatically splitting tablets/buckets to handle larger data volumes under the shared-data architecture.
* Multi-Node Tablet Writes: An enhancement specifically tagged for 4.2 (#79405) allows multiple compute nodes to write to the same tablet in shared-data mode, increasing ingestion throughput.
* Sub-second Data Freshness: Further optimizations to the real-time storage engine.
* AI & Vector Search:
* Native Vector Index: Built-in vector indexing to support AI-driven similarity search workloads.
* Full-text Inverted Index: Improvements to the built-in inverted index for faster text search.
* Performance:
* Global Late Materialization: Optimizing query performance by delaying data reading until after filtering.
* Fine-grained Tablet Parallelism: Improving concurrency within a single tablet scan.
3. Bug Fixes
Specific bug fixes for 4.2 will be finalized upon its release. Currently, the StarRocks team is actively addressing stability issues in the 4.1.x patch line, which will be rolled into 4.2. Common focus areas include:
* Memory Safety: Robustness in long-string and JSON processing scenarios.
* Catalog Stability: Fixing OAuth2/token-refresh issues for Iceberg REST catalogs.
* Query Correctness: Fixing edge cases in ARRAY/MAP constructors and complex aggregate pushdowns.
You can follow the live progress of version 4.2 features on the StarRocks 2026 Roadmap (GitHub #67632).
References
• release_notes/release-4.0.md
• release_notes/release-4.1.md
• release_notes/release-3.5.md
References
• release_notes/release-4.0.md
• release_notes/release-2.4.md
• release_notes/release-3.3.mdSai
09/23/2026, 3:05 PMRocky
09/23/2026, 3:05 PMJAVA_OPTS within fe.conf.
Recommendation for Upgrading:
If you plan to move to 4.2 once it is released, you should prepare your environment by ensuring your nodes are ready to run JDK 21. If you are currently on v3.x or 4.0/4.1, you should already be on JDK 17 at a minimum.
References
• release_notes/release-3.5.md
• deployment/upgrade.md
• release_notes/release-3.3.md
References
• release_notes/release-3.5.md
• deployment/upgrade.md
• release_notes/release-3.3.md
References
• release_notes/release-3.5.md
• developers/type_checker_xml_configuration.md
• deployment/environment_configurations.md
• deployment/upgrade.md
• sql-reference/sql-functions/JAVA_UDF.md
• release_notes/release-3.3.md
• release_notes/release-4.0.mdSai
09/23/2026, 3:06 PMRocky
09/23/2026, 3:07 PMpom.xml and build.gradle to target Java 21.
* Runtime Minimum: Raising the MIN_JDK_VERSION check in start_fe.sh to 21.
* Ecosystem Alignment: Ensuring compatibility with modern versions of Hadoop (3.4.x), Spark (3.5.x), and Iceberg (1.10.x) which fully support JDK 21.
* FE Improvements: Preparing the FE for future performance optimizations using Virtual Threads (JEP 444), which is a core feature of JDK 21.
Important Note for 4.1 Users:
If you are currently on version 4.1.x, you can continue using JDK 17. However, once you plan to upgrade to v4.2, you will be required to install JDK 21 on your FE and BE nodes to meet the new baseline requirements.
References
• deployment/deployment_prerequisites.md
• release_notes/release-3.5.md
• Version Release Guide - StarRocks Docs
• GitHub Feature #74519: Upgrade Java baseline to JDK 21 (v4.2)
References
• deployment/deployment_prerequisites.md
• release_notes/release-3.5.md
• release_notes/release-3.3.md
• release_notes/release-4.0.md
• release_notes/release-4.1.md
• GitHub Issue #74519: [Feature] Upgrade Java baseline from JDK 17 LTS to JDK 21 LTS (v4.2)
• GitHub Issue #67632: StarRocks 2026 Roadmap
References
• deployment/environment_configurations.md
• deployment/deployment_prerequisites.md
• release_notes/release-3.5.md
• faq/Deploy_faq.md
• sql-reference/sql-functions/JAVA_UDF.md
• release_notes/release-3.3.mdSai
09/23/2026, 3:11 PMRocky
09/23/2026, 3:12 PMstarrocks/fe-ubuntu, starrocks/be-ubuntu, etc.) are self-contained and will come pre-bundled with JDK 21 starting with version 4.2.
* Action: Simply update the image tag in your StarRocksCluster CRD or values.yaml file to 4.2.x.
2. Update the StarRocks Operator First
Before upgrading the cluster to 4.2, ensure your StarRocks Operator is up to date (recommended version v1.10.0 or later). The operator manages the rolling upgrade process and ensures that new Java parameters or environment variables are correctly passed to the new pods.
3. Review JAVA_OPTS in your YAML
If you have customized the JAVA_OPTS environment variable in your values.yaml or StarRocksCluster specification, review them:
* Remove Old Flags: Ensure there are no legacy garbage collection flags (like CMS) that were deprecated in newer Java versions.
* Keep Add-Opens: If you use Java UDFs or External Catalogs (Iceberg, Glue, etc.), you must ensure the following flags remain in your be.conf or `JAVA_OPTS`:
bash
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
Note: The official 4.2 images will likely include these by default, but double-check if you override them.
4. EKS-Specific Considerations
* IAM Roles for Service Accounts (IRSA): If you use S3-backed storage (Shared-Data) or Glue Catalogs, your IRSA configurations remain the same. Java 21 is fully compatible with the AWS SDKs used by StarRocks.
* Graviton (ARM) Nodes: If you are using Graviton-based instances (e.g., m7g, r7g), StarRocks 4.2 will continue to provide ARM64-compatible images with JDK 21.
* Storage (EBS/EFS): No changes are required to your Persistent Volume Claims (PVCs) for the Java upgrade.
5. Recommended Upgrade Step
As with any major version or baseline change, it is recommended to test the upgrade in a staging EKS namespace first. Use the Operator's rolling upgrade capability to monitor the health of one FE/BE at a time.
References
• deployment/sr_operator.md
• deployment/helm.md
• administration/management/FE_configuration.md
• GitHub Issue #74519: [Feature] Upgrade Java baseline from JDK 17 LTS to JDK 21 LTS (v4.2)
References
• deployment/sr_operator.md
• deployment/helm.md
• administration/management/FE_configuration.md
• deployment/environment_configurations.md
• deployment/deployment_prerequisites.md
• GitHub Issue #74519: [Feature] Upgrade Java baseline from JDK 17 LTS to JDK 21 LTS (v4.2)
• StarRocks on EKS - Best Practices Guide
• release_notes/release-3.5.md
References
• deployment/sr_operator.md
• deployment/helm.md
• quick_start/helm.mdTravis (YaaaDingus)
09/23/2026, 4:29 PMRocky
09/23/2026, 4:29 PMsql
SHOW PROC '/compactions';
* What to look for: If the list is empty or tasks stay in the queue (in_queue_sec is high), your BEs are not picking up new tasks.
* Shared-data (Cloud Native): Check information_schema.be_cloud_native_compactions for failure reasons.
2. Check Disk Space (The "High Water Mark")
StarRocks will stop compaction if a disk reaches a certain usage threshold to prevent full disk failure.
sql
SHOW PROC '/be_nodes';
* Check the UsedPct column. If any disk is above 90-95% (controlled by storage_flood_stage_usage_percent in be.conf), compaction may be disabled automatically. You must free up space or add nodes to resume.
3. Increase Compaction Threads
If the backlog is large, the default number of threads might be too low to clear it quickly. You can increase these dynamically (no restart required for most):
For Duplicate/Aggregate/Unique Key tables:
sql
-- Increase threads per disk (Default is usually 1 or 2)
ADMIN SET FRONTEND CONFIG ("cumulative_compaction_num_threads_per_disk" = "4");
ADMIN SET FRONTEND CONFIG ("base_compaction_num_threads_per_disk" = "2");
For Primary Key tables:
Primary Key compaction is more memory-intensive.
bash
# In be.conf or via curl
update_compaction_num_threads_per_disk = 4
4. Check for "Stuck" Rowsets in BE Logs
If a specific tablet has a corrupted rowset or a version gap, compaction might fail repeatedly. Search your BE logs (<http://be.INFO|be.INFO>) for error messages:
bash
grep -i "compaction" <http://be.INFO|be.INFO> | grep -i "fail"
Common issues include version already exists or rs_id not found. If you see a specific tablet ID failing repeatedly, it may need manual repair.
5. Manually Trigger Compaction (Primary Key Tables)
If you are using Primary Key tables, you can try to force a compaction on a specific table to see if it makes progress:
sql
-- This syntax is for PK tables to trigger manual compaction
ALTER TABLE <table_name> OPTIMIZE;
6. Verify Memory Limits
Since you recently resolved OOM issues, check if your compaction_mem_limit is set too low. If compaction tasks require more memory than allowed, they will be canceled.
* Check compaction_max_memory_usage_percent in be.conf (default is usually 80-100% of the process limit, but check if it was manually lowered during your OOM troubleshooting).
Summary Checklist:
1. Is Disk Usage < 90%?
2. Are there active tasks in SHOW PROC '/compactions'?
3. Are there "fail to compaction" errors in <http://be.INFO|be.INFO>?
4. Have you increased cumulative_compaction_num_threads_per_disk?
References
• administration/management/compaction.md
• faq/loading/Loading_faq.md
• administration/management/BE_configuration.md
References
• administration/management/compaction.md
• faq/loading/Loading_faq.md
• knowledge/admin_and_operation/shared_data_compaction.md
• administration/management/BE_configuration.md
• administration/management/FE_configuration.mdRocky
09/23/2026, 4:29 PMTravis (YaaaDingus)
09/23/2026, 4:33 PMALTER TABLE table_name COMPACT;, I don't see the compaction operation even get listedRocky
09/23/2026, 4:34 PMALTER TABLE ... COMPACT command isn't appearing in SHOW PROC '/compactions', or scores aren't dropping, here is how to troubleshoot:
1. Why ALTER TABLE ... COMPACT might not show up
If the command returns "OK" but nothing appears in `SHOW PROC '/compactions'`:
* Version Compatibility: Manual compaction for Shared-data was introduced in v3.3.0. If you are on an older version, the SQL syntax might be accepted but not implemented for cloud-native tables.
* Minimum Score Threshold: By default, StarRocks won't trigger a compaction task if the partition's score is below the threshold set by lake_compaction_score_selector_min_score (default is 10). If your "high" score is actually around 10-15, the FE might decide it's not worth the resource cost.
* Check the FE Leader Log: Search the fe.log on the Leader FE for the table name or "compaction" immediately after running the command. It will often log why a task was skipped (e.g., "no partition to compact" or "already has a running transaction").
2. Why scores "slightly improve" but stay high
Compaction in StarRocks is size-tiered. This means it groups files of similar sizes together.
* The "Large File" Wall: If you have one massive data file and 20 tiny ones, StarRocks will merge the 20 tiny ones into a medium one. Your score will drop (e.g., from 21 down to 2), but those medium files won't merge with the massive one until you have enough "medium" files to trigger a larger tier merge.
* Singleton Delta Limit: The parameter max_cumulative_compaction_num_singleton_deltas (default 100-500) limits how many files can be merged in one go. If you have 2,000 versions, one compaction run will only clear 500, requiring 4 separate runs to fully clear the backlog.
3. Tuning for a Deep Clean
Since your disks are empty and you have no load, you can force the system to be more aggressive:
On the FE (Global settings):
sql
-- Allow more concurrent tasks (default is usually #CNs * 16)
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "100");
-- Lower the threshold to allow cleaning even low-score partitions
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "0");
On the CN/BE (via update_configs or be.conf):
sql
-- Increase the number of threads per node
UPDATE information_schema.be_configs SET VALUE = "16" WHERE name = "compact_threads";
-- Speed up large backlogs by merging more files per task
UPDATE information_schema.be_configs SET VALUE = "1000" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
4. Advanced Debugging Commands
To see exactly what the FE thinks about your partitions and why it might be ignoring them, run:
sql
-- See exactly which partition has the highest MaxCS
SELECT * FROM information_schema.partitions_meta
WHERE TABLE_NAME = 'your_table_name'
ORDER BY MAX_CS DESC;
-- Look for hidden errors in recent cloud-native tasks
SELECT * FROM information_schema.be_cloud_native_compactions
WHERE STATUS != 'OK'
ORDER BY START_TIME DESC LIMIT 10;
Recommendation: If the score is stuck and manual SQL fails, try a Base Compaction specifically, which forces a merge into the base version:
sql
ALTER TABLE <table_name> BASE COMPACT;
References
• administration/management/compaction.md
• knowledge/admin_and_operation/shared_data_compaction.md
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
• administration/management/BE_configuration.mdRocky
09/23/2026, 4:34 PMTravis (YaaaDingus)
09/23/2026, 8:26 PMRocky
09/23/2026, 8:26 PMmax_cumulative_compaction_num_singleton_deltas. Your compaction is hitting the maximum allowed segments per task.
* `write_segment_count: 0`: No new data files were written to S3.
* score_after == `score_before`: The "MaxCS" (Max Compaction Score) did not budge.
This pattern typically happens for one of three reasons in StarRocks Shared-data clusters:
1. The "Size-Tiered" Trap
StarRocks groups files into "tiers" based on size (e.g., <1MB, 1-10MB, 10-100MB).
* The Issue: Compaction only merges files within the same tier. If you have 1,000 tiny files, it will merge 500 of them into one "medium" file.
* Why the score doesn't change: The Compaction Score is often dominated by the tier with the most files. If you merge 500 tiny files into 1 medium file, you still have 500 tiny files left in that tier. The MaxCS remains high because the "backlog" in that specific size group is still there.
* Why write_segment_count is 0: If you are using Primary Key tables and the compaction task only processed delete bitmaps or found that all rows in those 500 segments were already deleted/overwritten, it may result in 0 new data segments.
2. Primary Key Table: Persistent Index Issues
If this is a Primary Key table, compaction requires the "Persistent Index" to be healthy.
* If the index is corrupted or the CN node running the compaction is under extreme memory pressure (which you mentioned having recently), the compaction task might "finish" without actually committing the merge to storage.
* Check: Run SHOW INDEX FROM table_name; and check the BE logs for fail to load index or index checksum mismatch.
3. The "Max Version" Limit
If a single tablet has an extremely high version count (near 1,000), and the compaction task is limited to merging 500 (max_cumulative_compaction_num_singleton_deltas = 500), it only solves half the problem. If the task fails to commit for any reason (e.g., S3 timeout during the final commit phase), the score won't refresh.
──────────
Recommended Actions to "Force" Progress:
1. Increase the "Bulk" merge limit (CN side)
Since you have a score of 1000, you need to merge more than 500 files at once to see the score drop significantly.
sql
-- Allow a single task to merge up to 1000 files instead of 500
UPDATE information_schema.be_configs SET VALUE = "1000" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
UPDATE information_schema.be_configs SET VALUE = "1000" WHERE name = "lake_pk_compaction_max_input_rowsets";
2. Check for "Stuck" Transactions
Compaction is a transaction. If there is a "stale" or "failed" transaction holding a lock on those rowsets, new compactions will do the work but fail to "Publish" the result.
sql
-- Check for any "PREPARED" or "COMMITTED" transactions that are very old
SHOW PROC '/transactions';
3. Inspect the BE/CN Logs for "Write" Errors
Since write_segment_count is 0, search your CN/BE logs for the TxnID from your SHOW PROC '/compactions' output:
bash
grep "TxnID_from_profile" <http://be.INFO|be.INFO>
Look specifically for:
* fail to upload to s3
* tablet is under compaction
* too many versions
4. Temporary "Aggressive" FE Selection
Force the FE to pick this table more often by lowering the selection threshold:
sql
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "0");
Question for you: Is this a Primary Key table, and did you recently perform a massive DELETE or UPDATE operation? This often causes the "0 write" behavior because the compaction is effectively just cleaning up deleted records.
ReferencesRocky
09/23/2026, 8:26 PMTravis (YaaaDingus)
09/24/2026, 3:06 PMWayne Gibson
10/09/2026, 5:15 PMRocky
10/09/2026, 5:16 PMMERGE INTO on Iceberg V2: This is a major update allowing Merge-On-Read (MOR) operations directly on Iceberg V2 tables. This enables more efficient updates and deletes for lakehouse workloads without requiring a full table rewrite.
* Iceberg Row-Delta Commit Path: Integration of a native commit path for row-level changes in Iceberg.
2. AI & Vector Search
* Vector Search Framework: Significant updates to the internal vector search framework are slated for 4.2, improving the interface for AI-driven applications and similarity searches.
3. Observability & Management
* JSON Audit Logs: A new FE configuration audit_log_json_format allows StarRocks to output audit logs in structured JSON, making it easier to integrate with modern logging stacks like ELK or Datadog.
* Native Linux Packaging: Official support for Debian (.deb) and RPM packages is being introduced to simplify standard Linux installations (replacing the manual tarball extraction method for some users).
* Online `OPTIMIZE TABLE`: Enhancements to the ALTER TABLE ... OPTIMIZE command to reduce write-blocking during background maintenance tasks.
4. System & Performance
* StarOS 4.2: The underlying cloud-native storage management layer (StarOS) is being upgraded to version 4.2, which includes optimizations for shared-data cluster stability and S3 object management.
* CTE Optimization: New session variables to control the "force reuse" of Common Table Expressions (CTEs) to speed up the optimizer for extremely complex queries.
How to verify
Since 4.2 is nearing release, you can see the latest progress by:
1. Checking the StarRocks/starrocks GitHub Issues with the label version:4.2.
2. Looking at the branch-4.2 or main branch code in the official repository.
3. Monitoring the official StarRocks Blog for the upcoming "StarRocks 4.2: What's New" announcement.