Nicholas Jordan
07/08/2026, 3:29 PMRocky
07/08/2026, 3:29 PMJAVA_OPTS provided in the configuration files, as they are tuned for modern JDKs (G1GC).
Upgrading from 3.5 to 4.1
StarRocks 4.1 is a production-ready release ("Built for Production"), but there are important considerations for the upgrade path:
1. Upgrade Path: For compatibility and safety, StarRocks strongly recommends consecutive minor version upgrades. To move from 3.5 to 4.1, you should follow this sequence:
* v3.5.x → v4.0.x → v4.1.x
* Do not skip the 4.0 minor version, as this ensures metadata and system behavior transitions are handled correctly.
2. Stability Status:
* v3.5.x is currently designated as the Stable/LTS (Long-Term Support) branch.
* v4.1.x is the Latest feature-rich version. It is stable for production use but contains more recent features (like range-based tablet auto-splitting and V2 schema evolution) that may have a different maturity level compared to the battle-tested 3.5 line.
3. Key Upgrade Notes:
* JDK: Ensure your environment is already on JDK 17+ before starting the upgrade from 3.5.
* Metadata: Always back up your FE metadata (meta directory) before performing any minor or major version upgrade.
* Check Prerequisites: Review the specific release notes for both 4.0 and 4.1 to check for any deprecated parameters or mandatory configuration changes (e.g., JAVA_OPTS changes for JNI or external catalogs).
References
• release_notes/release-3.5.mdNicholas Jordan
07/08/2026, 3:31 PMRocky
07/08/2026, 3:32 PMNicholas Jordan
07/08/2026, 3:34 PMRocky
07/08/2026, 3:34 PMpartition field within the updateStrategy.
The StarRocks Operator manages BE nodes using a StatefulSet. By utilizing the standard Kubernetes StatefulSet "Partition" strategy, you can update only a subset of your BE pods to a new image version to verify stability before completing the rollout.
How to Implement a Canary Rollout
You can configure this in your StarRocksCluster Custom Resource (CR). For example, if you have 5 BE nodes and want to update only the last one (index 4) as a canary:
yaml
spec:
starRocksBeSpec:
image: starrocks/be-ubuntu:4.1.1 # The new image version
updateStrategy:
type: RollingUpdate
rollingUpdate:
partition: 4 # Only pods with index >= 4 will be updated
When you apply this configuration:
1. Kubernetes will update only be-4.
2. be-0 through be-3 will remain on the old image.
3. Once you are satisfied with the canary's performance, you can set partition: 0 to roll out the update to the remaining nodes.
How BE Nodes Handle the Rollout
StarRocks is designed for rolling upgrades, so BE nodes handle this "mixed version" state well:
* Version Compatibility: StarRocks BEs are designed to be backward compatible with FEs. During a canary, your newer BEs will still correctly communicate with the existing FE leader.
* Data Availability: In a Shared-nothing architecture, the Operator restarts BE pods one at a time. As long as your tables have a replication factor of 3, the cluster remains fully available even while one node is being upgraded.
* Tablet Balancing: To prevent the cluster from trying to "repair" data while a node is briefly down during the image swap, it is recommended to disable tablet balancing before starting the rollout:
sql
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "true");
(Remember to re-enable these once the full rollout is complete and all nodes are Alive.)
Key Considerations
1. Upgrade Order: Always upgrade BEs first, then FEs. A newer BE can work with an older FE, but an older BE may not understand metadata from a newer FE.
2. Shared-Data (Cloud-Native): If you are using the shared-data architecture (Compute Nodes / CN), canary rollouts are even lower risk because the nodes are stateless and do not require tablet redistribution.
3. Graceful Shutdown: The StarRocks Operator uses preStop hooks to ensure BEs attempt to finish ongoing tasks before shutting down, which minimizes query failures during the rollout.
References
• deployment/sr_operator.md
• deployment/upgrade.md
References
• deployment/upgrade.md
• deployment/downgrade.mdNicholas Jordan
07/08/2026, 3:37 PMRocky
07/08/2026, 3:38 PMbe-0) fails its Readiness Probe with the new image, Kubernetes will not proceed to update the next pod.
* The upgrade effectively "halts." Your cluster will continue running with a mix of versions: the first node will be in a CrashLoopBackOff or Unready state, while the remaining nodes remain on the old, healthy image.
2. Monitoring Health
As of Operator v1.8.7, you can fine-tune how sensitive this detection is by setting failure thresholds in your StarRocksCluster YAML:
yaml
spec:
starRocksBeSpec:
# How long to wait/fail before the pod is considered "dead"
readinessProbeFailureSeconds: 60
livenessProbeFailureSeconds: 60
If the BE doesn't become "Ready" within these windows, the rolling update stops automatically.
3. How to Rollback Manually
If you detect a bug in your canary or if the rolling update has halted, you must trigger a rollback manually. There are two ways:
* Option A (Recommended): Update the image field in your StarRocksCluster CR back to the previous stable version. The Operator will see the change and start a new rolling update to revert the pods.
* Option B: Use the standard Kubernetes command:
bash
kubectl rollout undo statefulset/<cluster-name>-be
4. Achieving Fully Automated Rollback
To get the "detect failure -> trigger rollback" behavior you're looking for without manual intervention, you would typically use an external Continuous Delivery (CD) tool alongside the Operator:
* ArgoCD: Can be configured with "AnalysisTemplates" to monitor the health of the StarRocksCluster resources and automatically perform a rollout undo if success criteria aren't met.
* FluxCD: Offers similar automated rollbacks based on health checks.
* CI/CD Scripts: A simple kubectl rollout status command in your deployment pipeline can be used to block the pipeline and trigger a revert if the timeout is reached.
Summary Recommendation
To minimize risk without a native auto-rollback feature:
1. Use the Canary (Partition) method I mentioned previously. This ensures only 1 node is "sacrificed" to test the new image.
2. Monitor the BE Status using kubectl get starrockscluster. If BESTATUS stays in reconciling for too long, investigate the logs and manually revert the image in the YAML.
References
• quick_start/helm.md
• administration/Meta_recovery.mdNicholas Jordan
08/05/2026, 6:28 PMRocky
08/05/2026, 6:29 PMregexp_extract, regexp_replace, or random() are built into the StarRocks core engine.
* Where they reside: These are implemented in C++ within the *Backend (BE)*'s vectorized execution engine.
* How they are "loaded": They are not loaded dynamically from external files; they are part of the compiled BE binary.
* Execution Flow:
1. The FE (Frontend) parses the SQL, validates the function signature, and includes the function in the physical execution plan.
2. The BE receives the plan and executes the function logic directly on the vectorized data blocks using the pre-compiled C++ code.
2. Random Distribution (Bucketing)
"Random Distribution" is a data distribution strategy (Random Bucketing) rather than a simple SQL function.
* Where it resides: The logic is shared between the FE (for metadata and planning) and the BE (for execution).
* How it works:
* At Table Creation: The FE stores the distribution type in its metadata (BDB JE).
* During Data Load: When you load data, the BE (specifically the Load Channel/Sink) applies the random distribution logic. Instead of hashing a key, it uses a random number generator to pick a target bucket for each batch of data. This logic is compiled into the BE's data-routing components.
3. User-Defined Functions (UDFs)
If you are referring to functions that you load into the system, these follow a different path:
* Java UDFs: These are uploaded to a repository (usually HDFS) and then cached locally on each BE in the directory defined by local_library_dir (defaulting to ${STARROCKS_HOME}/lib/udf).
* Python UDFs: These are "loaded" into separate Python worker processes managed by the BE. You specify the environment path using the python_envs configuration in be.conf.
Summary
* Built-in Functions (Regex, etc.): Compiled into the BE C++ binary.
* Distribution Logic: Compiled into both FE (Java) and BE (C++).
* UDFs: Loaded dynamically from the UDF library directory on the BE at runtime.
References
• sql-reference/sql-functions/Python_UDF.md
• administration/management/BE_configuration.md
• sql-reference/sql-functions/hive_bitmap_udf.md
• sql-reference/sql-functions/README.md
• sql-reference/sql-functions/JAVA_UDF.mdNicholas Jordan
08/05/2026, 6:31 PMRocky
08/05/2026, 6:31 PMNo matching function with signature: regexp_count(...) occurs because the regexp_count function was introduced in StarRocks version 4.0.
If you are running any version prior to 4.0 (such as 2.5, 3.1, or 3.2), the Frontend (FE) does not have this function in its built-in library, which is why the SQL analyzer fails to recognize the signature.
How to resolve this
1. Upgrade to StarRocks 4.0+
The function is officially supported as a built-in starting from version 4.0. Upgrading your cluster will make the function available immediately.
2. Workaround for older versions (3.x)
If you cannot upgrade yet, you can achieve the same result by using other regex functions.
Option A: Using regexp_extract_all and array_length
If your version supports regexp_extract_all (introduced in 3.1), you can count the elements in the resulting array:
sql
SELECT array_length(regexp_extract_all(str, regex))
FROM tsc_invalid;
Option B: Using regexp_replace (Less efficient but compatible with 2.x/3.x)
You can replace all matches with a specific single character and compare lengths, or use a trick with length and `replace`:
sql
-- Replace all non-matching parts with nothing, then count what's left
-- Note: This is complex with regex; usually Option A is better.
Why you saw it in the docs
The StarRocks documentation usually defaults to the "Latest" version. If you are looking at the docs for 4.0 or "Latest" while running an older version, you will see functions that haven't been backported to your specific release. You can check your current version by running:
sql
SELECT version();
References
• sql-reference/sql-functions/like-predicate-functions/regexp_count.md
• sql-reference/sql-functions/like-predicate-functions/regexp_extract.md
• sql-reference/sql-functions/string-functions/regexp_split.md
• sql-reference/sql-functions/like-predicate-functions/regexp.md
References
• sql-reference/sql-functions/like-predicate-functions/regexp_count.md
• sql-reference/sql-functions/like-predicate-functions/regexp_extract.md
• sql-reference/sql-functions/like-predicate-functions/regexp.md
• sql-reference/sql-functions/string-functions/regexp_split.md
• sql-reference/sql-statements/table_bucket_part_index/SELECT.md
• sql-reference/sql-functions/aggregate-functions/count.mdNicholas Jordan
08/05/2026, 6:37 PMRocky
08/05/2026, 6:37 PMtest_sql_cases). Specifically, the test test_agg_state_table_with_all_functions failed because the actual query results did not match the expected results.
This specific test is related to the Generic Aggregate Function States feature introduced in StarRocks v3.4.0.
Why this failure occurs
The error Element counts were not equal means the query returned a different number of rows or array elements than what the test expected. In the context of "Agg State" functions, this usually happens for one of the following reasons:
1. Version Mismatch
The "Generic Aggregate State" feature is rapidly evolving. If you are running a test suite from a newer StarRocks version (e.g., the main branch or v4.0) against a cluster running an older version (e.g., v3.3 or an early v3.4 release), the test will fail.
* v3.4.0+: Supports generic states for most built-in functions.
* Pre-v3.4.0: Only supports a limited set (SUM, MIN, MAX, BITMAP_UNION, HLL_UNION).
* Result: If the test tries to use avg_state() or array_agg_state() on a version that doesn't support them, the results will be empty or NULL, causing the "Element counts" mismatch.
2. Unsupported Functions
Even in v3.4+, several complex aggregate functions do not yet support generic states. These are internally tracked in UNSUPPORTED_AGG_STATE_FUNCTIONS. Common ones include:
* GROUP_CONCAT
* WINDOW_FUNNEL
* APPROX_TOP_K
If the "all functions" test includes one of these and your specific sub-version hasn't implemented it yet, the state column will not aggregate correctly.
3. Environment/Configuration
The test might be sensitive to specific BE configurations. Generic aggregate states are stored as intermediate binary data. If there is a mismatch in how Nullable columns are handled or if a specific BE variable (like enable_aggregate_state_compression) is toggled differently than the test expects, the output counts can differ.
How to troubleshoot
1. Check Version: Run SELECT version();. Ensure your cluster is at least v3.4.0.
2. Verify the SQL: Locate the SQL being executed by the test (usually in a .sql or .yaml file in the StarRocks source code under test/sql/). Run the query manually.
3. Check for NULLs: If the query returns fewer elements than expected, check if the _state or _union columns are returning NULL.
sql
-- Example check
SELECT avg_union(avg_state(col)) FROM your_table;
4. Align Test Suite: Ensure your test code branch matches your StarRocks binary version exactly (e.g., use the branch-3.4 test suite for a v3.4 cluster).
Summary of Feature Support
Feature | Version
Basic Agg Tables | v2.x and earlier
Generic Agg States (_state, _union) | v3.4.0+
regexp_count support | v4.0+
If you are seeing this while trying to use regexp_count (from your previous question), that is likely the culprit—`regexp_count` is a v4.0 feature and won't work in the aggregate state framework of v3.4.
References
• table_design/table_types/aggregate_table.md
• introduction/maturity.md
• sql-reference/sql-functions/aggregate-functions/avg.md
• sql-reference/sql-functions/aggregate-functions/sum.md
• sql-reference/sql-functions/aggregate-functions/any_value.md
References
• table_design/table_types/aggregate_table.mdNicholas Jordan
08/05/2026, 6:39 PMRocky
08/05/2026, 6:39 PMtest_sql_cases.py framework). When you see [FAILURE] test_sql_cases.TestSQLCases, it means the actual output from your StarRocks cluster did not match the "expected" result defined in the test's .sql or .yaml files.
Here is a breakdown of why these specific cases are failing:
1. test_approx_top_k_with_partition
* The Error: '' != [] : expected empty but got ([])
* Reason: This is usually a result format mismatch.
* The approx_top_k function returns an ARRAY of structs.
* When a query has no results, it should return an empty array [].
* The test case expectation was likely recorded as an empty string '' or a different empty format.
* Common Cause: This often happens if you are running a newer version of the test suite against an older StarRocks binary (or vice versa) where the default representation of an empty array changed in the shell output.
2. test_async_mv (Asynchronous Materialized View)
* The Error: Element counts were not equal: First has 1, Second has 0: '1\t3' ... act: ['']
* Reason: This is a timing/refresh issue.
* Asynchronous MVs depend on a background task to refresh data. The test expects the row 1\t3 to be in the MV (tva).
_ The actual result [''] (empty) indicates that the query SELECT _ FROM tva returned no data.
* Common Cause: The test suite likely queried the MV before the background refresh task finished. This is a known "flaky" behavior in distributed environments. You can often resolve this by manually checking the MV status:
sql
SHOW MATERIALIZED VIEWS LIKE 'tva';
-- Check if 'last_refresh_state' is SUCCESS
3. test_agg_state_table_with_all_functions
* The Error: Element counts were not equal
* Reason: This test validates Aggregate State (Generic Aggregate Functions), a feature that allows you to store intermediate aggregation states (e.g., avg_state).
* Common Cause: This failure often points to a version or configuration mismatch. This feature (introduced in v3.4) has several "unsupported" functions. If the test suite includes a function that your current BE version doesn't support in an aggregate state table, the row count or state data will be inconsistent.
How to Fix/Troubleshoot:
1. Check Cluster Health: Ensure all Backends (BEs) are alive (SHOW BACKENDS;). If some BEs are down, queries will return partial or empty results, causing these "count mismatch" errors.
2. Verify Version: Run SELECT version();. Ensure your StarRocks binary matches the version of the test suite you are running. If you pulled the main branch of the StarRocks repository but are running v3.2 binaries, these tests will fail.
3. Manual Refresh: For the async_mv failure, try running the refresh manually and see if the data appears:
sql
REFRESH MATERIALIZED VIEW tva;
4. Replication Num: In single-node dev environments, many tests fail because they expect replication_num = 3. Ensure your cluster has at least 3 BEs or that you have set admin set frontend config ("default_replication_num" = "1");.
If you are running these tests as part of a StarRocks contribution, it is recommended to check the logs in fe/log/fe.log and be/log/be.INFO for any specific "No alive nodes" or "Memory limit exceeded" errors that occurred during the test run.
References
• using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.mdNicholas Jordan
08/05/2026, 6:42 PMRocky
08/05/2026, 6:42 PMregexp_count fails
While regular expression operators (like REGEXP or RLIKE) are built-in for all versions, the specific function regexp_count was only introduced in StarRocks 4.0.
Since you are on 3.5.19, the SQL analyzer does not find a function named regexp_count in the system library.
* Fix: Use array_length(regexp_extract_all(str, pattern)) as a workaround in 3.5.x.
* Official Support: Upgrade to 4.0+ to use regexp_count directly.
──────────
2. Regression Test Failures
The failures you listed (test_agg_state_table_with_all_functions, etc.) indicate you are likely running the StarRocks regression test suite (Python-based test_sql_cases).
Failure 1: test_agg_state_table_with_all_functions
* Reason: This test validates "Generic Aggregate States" (intermediate binary storage for aggregations).
* Why it fails in 3.5.19: If you are running the test code from the main or 4.0 branch against a 3.5.19 cluster, the test will try to use functions in an Aggregate State table that your 3.5 cluster doesn't yet support (e.g., regexp_count_state or newer window function states).
* Check: Ensure your test suite branch matches your cluster version.
Failure 2: test_approx_top_k_with_partition ('' != [])
* Reason: This is a format mismatch.
* Details: The approx_top_k function returns an ARRAY. The test suite expects an empty string '' for empty results, but your version is returning an empty array []. This is a common "flaky" failure in the regression suite when the FE's default output format for empty complex types changes between versions.
Failure 3: test_async_mv (Row count mismatch)
* Reason: This is almost always a timing issue with asynchronous refreshes.
* Details: The test queries the materialized view before the background refresh task has finished.
* Fix: Manually trigger REFRESH MATERIALIZED VIEW tva; and verify with SHOW MATERIALIZED VIEWS before running the selection.
Summary
If you are developing or testing, ensure your test suite branch and cluster version are aligned. Most of these errors stem from trying to run "Version 4.0" tests on a "Version 3.5" cluster.
References
• release_notes/release-4.0.md
• release_notes/release-3.5.md
• sql-reference/sql-functions/like-predicate-functions/regexp_count.md
• table_design/table_types/aggregate_table.md
• using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md
References
• table_design/table_types/aggregate_table.mdNicholas Jordan
08/05/2026, 6:57 PMRocky
08/05/2026, 6:57 PM3.5.19 branch (or branch-3.5) are designed for that version. However, seeing these specific failures in that branch usually points to one of three things: environmental constraints, timing flakes, or a local workspace mismatch.
Here is why those "branch-specific" tests are likely failing on your 3.5.19 cluster:
1. Environmental Flakes (The test_async_mv failure)
The test_async_mv failure (First has 1, Second has 0) is a classic "flaky" test.
* Why it happens: Asynchronous Materialized Views rely on a background task to refresh. The test script sends a SELECT immediately after an insert. If your machine is under load or the FE background thread is a split-second slow, the MV will still be empty when the test queries it.
* Is it a bug? Not necessarily in the code, but a "timing flake" in the test suite itself.
2. Result Formatting (The test_approx_top_k failure)
The error '' != [] (expected empty string but got an empty array) usually happens because of the client or output configuration.
* The Mismatch: StarRocks 3.5 returns [] for an empty ARRAY type. If the test expectation was recorded using a different client (like an older MySQL client or a different test_sql_cases runner setting) that interprets empty arrays as empty strings, it fails.
* Branch Context: Sometimes, a patch is backported to a branch (like 3.5.19) but the corresponding "result file" (the .result or .yaml file the test checks against) is accidentally updated to a format that matches main (4.0) instead of the stable branch.
3. The "All Functions" Problem (test_agg_state_table_with_all_functions)
This test is particularly sensitive. It typically iterates through a list of functions that the Front End (FE) believes are supported for Aggregate State.
* The Conflict: If you have a mismatch between your FE and BE binaries (e.g., you updated the FE to 3.5.19 but one or more BEs are still on an older version), the FE will try to use a 3.5 feature that the BE doesn't recognize yet.
* Branch-Specific Bugs: Occasionally, a function is added to the "Supported" list in the FE code on branch-3.5, but the underlying BE implementation for that specific version has a corner-case bug.
How to verify your setup:
To ensure the tests match your cluster:
1. Check Binary Sync: Run SHOW BACKENDS\G and verify that the Version column for every BE matches your FE version exactly (3.5.19).
2. Verify Test Code Source: Ensure you aren't running the tests from a main branch checkout. In your terminal, run git branch inside the StarRocks source folder.
3. Manual Check: Take the SQL from the failing test_async_mv and run it manually in a MySQL client. If it returns the correct data after a few seconds, it confirms the "timing flake" theory.
If you are certain the environment and versions are perfectly aligned, these may be regressions specific to the 3.5.19 release, and reporting them on the StarRocks GitHub Issues with your fe.log would be very helpful to the benefit of great help to the maintainers.
References
• using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md
• administration/management/resource_management/query_queues.md
• table_design/table_types/aggregate_table.md
• introduction/maturity.md
• administration/management/resource_management/plan_reusage.md
ReferencesRocky
08/05/2026, 6:57 PM