http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> What is the step by step guide to u...
# questions-and-troubleshooting
m
@Rocky What is the step by step guide to upgrading a StarRocks shared nothing cluster version. I want to upgrade from 4.0.9 to the latest
r
To upgrade your StarRocks shared-nothing cluster from version 4.0.9 to the latest version (currently v4.1.x as of mid-2026), follow this step-by-step guide. Important Prerequisites * Upgrade Order: You must upgrade BEs and CNs first, and then FEs. * FE Order: Upgrade all Follower FEs before upgrading the Leader FE. * Backup: It is highly recommended to back up your metadata (the 
meta
directory of the FE) before starting. * Consecutive Upgrades: While you can skip patch versions (e.g., 4.0.9 to 4.1.1), always check the release notes for any specific "stop-over" versions or breaking changes. ────────── Step 1: Preparation (Disable Balancing) Before starting the upgrade, you must disable tablet balancing to prevent the cluster from trying to redistribute data while nodes are offline.
Copy code
sql
-- Disable tablet scheduling and balancing
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0");
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0");
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "true");
Step 2: Upgrade BE Nodes (Rolling Upgrade) Perform these steps on one BE node at a time. Ensure the node is back and "Alive" before moving to the next. 1. Stop the BE:
Copy code
bash
    cd <be_dir>/be
    ./bin/stop_be.sh
2. Replace Binaries: Replace the 
bin
and 
lib
directories with the new version.
Copy code
bash
    mv lib lib.bak && mv bin bin.bak
    cp -r /tmp/StarRocks-4.1.x/be/lib .
    cp -r /tmp/StarRocks-4.1.x/be/bin .
    # If using UDFs, copy them back
    cp -r lib.bak/udf lib/
3. Start the BE:
Copy code
bash
    ./bin/start_be.sh --daemon
4. Verify: Check the status in MySQL: 
SHOW PROC '/backends';
. Ensure 
Alive
is 
true
. Step 3: Upgrade CN Nodes (Optional) If your cluster uses Compute Nodes (CN), follow the same procedure as the BE upgrade, using 
stop_cn.sh --graceful
and 
start_cn.sh
. Step 4: Upgrade FE Nodes Upgrade Follower FEs first, then the Leader. 1. Stop the FE:
Copy code
bash
    cd <fe_dir>/fe
    ./bin/stop_fe.sh
2. Replace Binaries:
Copy code
bash
    mv lib lib.bak && mv bin bin.bak && mv spark-dpp spark-dpp.bak
    cp -r /tmp/StarRocks-4.1.x/fe/lib .
    cp -r /tmp/StarRocks-4.1.x/fe/bin .
    cp -r /tmp/StarRocks-4.1.x/fe/spark-dpp .
3. Start the FE:
Copy code
bash
    ./bin/start_fe.sh --daemon
4. Verify: Check the status: 
SHOW PROC '/frontends';
. Ensure 
Alive
is 
true
and the version is updated. 5. Repeat for all Followers, then finally the Leader. Step 5: Post-Upgrade (Re-enable Balancing) Once all nodes are upgraded and stable, re-enable balancing:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "10000");
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "500");
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");
ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "false");
Critical Notes for v4.1+ * Downgrade Restriction: If you upgrade to v4.1, you can only downgrade to v4.0.6 or later. Downgrading to versions earlier than 4.0.6 is not supported due to changes in data layout. * Behavior Changes: v4.1 introduces changes to how 
parse_json
handles exceptions and how Parquet timestamps are read. Review the official 4.1 release notes for specific impacts on your queries. References • deployment/upgrade.md • deployment/sr_operator.md • deployment/deploy_manually.md
m
@Rocky Where do I get the new version files? It wasnt mentioned in the docs
r
You can obtain the StarRocks binary distribution packages through the following three methods: 1. Official Download Page (Recommended) The easiest way is to use the official StarRocks Download Page. * Select the version (e.g., 4.1.x). * Select your Operating System (CentOS/Ubuntu). * Select your CPU Architecture (AMD64/x86 or ARM64). * Copy the download link or download the 
.tar.gz
file directly. 2. GitHub Releases You can find all versions, including pre-releases and older patches, on the StarRocks GitHub Releases page. * Scroll down to the version you want. * Under the Assets section, look for the file named 
StarRocks-<version>-<os>-<arch>.tar.gz
. 3. Direct Download via Command Line If you are working directly on your server, you can use 
wget
. The download URL follows this pattern:
Copy code
bash
# Example for version 4.1.1 on CentOS/x86
wget <https://releases.starrocks.io/starrocks/StarRocks-4.1.1-centos-amd64.tar.gz>
Replace 
4.1.1
with your desired version and 
centos
with 
ubuntu
if applicable. Important Note on Architecture * x86/AMD64: Standard binaries are readily available as 
.tar.gz
files. * ARM64: If you are running on ARM (e.g., AWS Graviton or Apple Silicon), StarRocks provides specific ARM binaries on the download page. If they are not listed for a specific version, you may need to prepare them using the StarRocks Docker image.
m
@Rocky How do I check whether my EC2 machine should use the CentOS or ubuntu release
r
To determine which StarRocks binary to download, you need to check the Operating System and the CPU Architecture of your EC2 instance. Step 1: Check the Operating System Run the following command in your terminal:
Copy code
bash
cat /etc/os-release
How to interpret the results: * Use the 
centos
release if: The output shows 
NAME="Amazon Linux"
, 
NAME="CentOS Linux"
, or 
ID_LIKE="centos rhel fedora"
. _ _Note: Both Amazon Linux 2 and Amazon Linux 2023 are RPM-based and should use the CentOS binary.* * Use the 
ubuntu
release if: The output shows 
NAME="Ubuntu"
or 
ID="ubuntu"
. Step 2: Check the CPU Architecture StarRocks provides different binaries for Intel/AMD and ARM (Graviton) processors. Run:
Copy code
bash
uname -m
How to interpret the results: * `x86_64`: Use the 
amd64
 binary. * `aarch64`: Use the 
arm64
 binary. ────────── Summary Table Based on your findings, choose the link that matches your environment: If 
os-release
says... | And 
uname -m
says... | Use this StarRocks Binary Amazon Linux / CentOS | 
x86_64
| 
StarRocks-x.x.x-centos-amd64.tar.gz
Amazon Linux / CentOS | 
aarch64
| 
StarRocks-x.x.x-centos-arm64.tar.gz
Ubuntu | 
x86_64
| 
StarRocks-x.x.x-ubuntu-amd64.tar.gz
Ubuntu | 
aarch64
| 
StarRocks-x.x.x-ubuntu-arm64.tar.gz
Example Download Command If you are on Amazon Linux 2 with an x86 processor (most common), you would download v4.1.x like this:
Copy code
bash
wget <https://releases.starrocks.io/starrocks/StarRocks-4.1.1-centos-amd64.tar.gz>
m
@Rocky Is it safe it I test upgrade 1 BE node first and see how it performs while the other stays on older version? v4.0.9 vs v4.1.3
r
Yes, it is safe and actually recommended to upgrade a single BE node first as a test. This is referred to in the StarRocks documentation as an "Upgrade Availability Test." However, there are a few technical nuances to keep in mind when running in a "mixed version" state (v4.0.9 and v4.1.3): 1. Why it is safe * Backward Compatibility: StarRocks is designed such that newer BE versions are backward compatible with older FE versions. The v4.1.3 BE will be able to process requests sent by your v4.0.9 FE. * Official Recommendation: The official upgrade guide explicitly suggests performing an "upgrade availability test on one of the... BEs before upgrading all nodes." 2. Critical Precautions for the Test While the cluster will be functional, you must follow these rules during the mixed-version period: * Keep Balancing Disabled: You must keep 
disable_balance = true
(and other balancing configs) active while the BEs are on different versions. This prevents the cluster from trying to move data between a v4.1 BE and a v4.0 BE, which could potentially lead to data format issues if there were underlying storage changes. * Limit the Test Duration: While safe for a test, do not leave the cluster in a mixed state indefinitely. Once you are satisfied with the performance of the upgraded BE, proceed to upgrade the remaining BEs and then the FEs. * New Features Won't Work: Any new features or query optimizations exclusive to v4.1 will not be active until the FEs are also upgraded to v4.1, as the FE is responsible for query planning. 3. How to "See How it Performs" Since you are testing for performance, keep in mind: * Check the Logs: After upgrading the first BE, monitor its 
<http://be.INFO|be.INFO>
and 
be.warn.log
for any unexpected errors. * Target Queries: Use 
EXPLAIN ANALYZE
or the StarRocks Query Profile to verify that queries hitting the v4.1 BE are executing correctly. * Monitor Metrics: Compare the CPU, memory, and IO usage of the v4.1 BE against the remaining v4.0 BEs using your monitoring dashboard (e.g., Grafana). Summary of the "Single-Node Test" Workflow 1. Disable balancing (via 
ADMIN SET FRONTEND CONFIG
). 2. Upgrade 1 BE to v4.1.3. 3. Verify it shows as 
Alive
in 
SHOW PROC '/backends';
. 4. Observe performance and stability for a few hours. 5. Proceed to upgrade the rest of the BEs, then the FEs, following the standard order. Warning on Downgrading: If the test fails and you need to go back, you can generally downgrade the BE. However, note that v4.1 introduces storage changes; while rolling back a single BE is usually safe if no metadata-level upgrade has occurred on the FEs, always ensure you have a metadata backup of your FEs before you started the whole process. References • deployment/downgrade.md • release_notes/release-4.0.md • deployment/upgrade.md • administration/management/BE_configuration.md
m
@Rocky I have upgraded my cluster to v4.1.3 . This is part of my table definition
Copy code
`deal_ids` array<varchar(65533)> NULL COMMENT "",
PRIMARY KEY(`received`, `auction_id`, `imp_key`)
PARTITION BY date_trunc('hour', received)
DISTRIBUTED BY HASH(`auction_id`) BUCKETS 128
ORDER BY(`received`, `auction_id`)
PROPERTIES (
"compression" = "LZ4",
"enable_persistent_index" = "true",
"fast_schema_evolution" = "true",
"partition_live_number" = "24",
"replicated_storage" = "false",
"replication_num" = "1"
);
I want to add the built-in index introduced in v4.1 to the 
deal_ids
column. My deal_ids are not striclty alphanumeric, containing dashes and underscores. To my knowledge, this means I should use parser='none' . I dont need substring lookup so dict_gram_num not required, more efficient index building. Also, I think I also need to pre-create the future partitions in advance so new partition creation at the turn of the hour doesn't affect index building? Give me the steps to do so
r
To add the v4.1 built-in inverted index to your 
deal_ids
column and manage your partitions, follow these steps: 1. Enable the Inverted Index Feature The "built-in" inverted index (GIN) is currently an experimental feature in the FE. You must enable it first:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("enable_experimental_gin" = "true");
2. Add the Index to the 
ARRAY
Column Since your 
deal_ids
contain underscores and dashes and you don't need substring lookups, using 
parser='none'
with the 
builtin
implementation is the correct choice. You do not need to set 
dict_gram_num
(it defaults to 
-1
, which disables the n-gram dictionary acceleration you don't need). Run the following command:
Copy code
sql
CREATE INDEX idx_deal_ids ON your_table_name (deal_ids) 
USING GIN (
    "parser" = "none", 
    "imp_lib" = "builtin"
) 
COMMENT 'Index for array elements';
Note: For 
ARRAY<VARCHAR>
columns, StarRocks builds the index on the individual elements of the array. 3. Monitor the Progress Adding an index to an existing table is an asynchronous operation. StarRocks will build the index for all historical data in the background. You can check the status using:
Copy code
sql
SHOW ALTER TABLE COLUMN;
Look for the 
State
column; it will show 
FINISHED
once the index is fully built for existing data. ────────── 4. Pre-creating Future Partitions Your table uses Expression Partitioning (
date_trunc('hour', received)
). While StarRocks automatically creates partitions when data arrives, you can manually pre-create them using the 
ALTER TABLE ... ADD PARTITION
syntax. Because 
date_trunc
behaves like Range Partitioning under the hood, you add partitions by specifying the upper bound (the start of the next hour). Example: Pre-creating the next 3 hours If the current time is 
2024-05-20 10:00:00
, you would run:
Copy code
sql
-- For the 11:00 slot
ALTER TABLE your_table_name ADD PARTITION p11 
VALUES LESS THAN ("2024-05-20 12:00:00");

-- For the 12:00 slot
ALTER TABLE your_table_name ADD PARTITION p12 
VALUES LESS THAN ("2024-05-20 13:00:00");
Why do this? * Avoid Performance Spikes: Manual creation prevents the small metadata lock/overhead that occurs when a high-concurrency load triggers an "auto-create" at the top of the hour. * Index Behavior: Once the 
CREATE INDEX
task is 
FINISHED
, any newly created partition (manual or automatic) will have the index structure created immediately as data is ingested. Pre-creating partitions doesn't change how the index is built, but it gives you more control over the cluster's workload at the hour turnover. Important Notes * Querying: To use the index on an 
ARRAY
column, use the 
ARRAY_CONTAINS
function or the 
MATCH
predicate. Since you used 
parser='none'
, 
MATCH
will behave like an exact match or a 
LIKE
filter on the array elements. * Replicated Storage: Ensure 
replicated_storage
is set to 
false
(which it is in your definition). In v4.1, the built-in index requires this for optimal performance. References • table_design/indexes/inverted_index.md • sql-reference/data-types/semi_structured/Array.md
m
@Rocky Can I not use the explain analyze trick to pre-create future partitions
r
Yes, you absolutely can use the 
EXPLAIN ANALYZE
 trick. In StarRocks, this is a recognized method for pre-creating partitions in tables using Expression Partitioning (like your 
date_trunc('hour', ...)
setup) without actually loading any data. How it works When you run 
EXPLAIN ANALYZE INSERT
, the Frontend (FE) calculates which partitions are needed for the data being "inserted." If those partitions don't exist yet, the FE creates them as part of the planning process. Because it's an 
EXPLAIN
statement, the actual data is never written to disk, and no transaction is committed. The SQL Command To pre-create partitions for the next 8 hours (to cover your hourly turn-over), run this:
Copy code
sql
EXPLAIN ANALYZE
INSERT INTO your_table_name (received)
SELECT DATE_ADD(NOW(), INTERVAL d hour)
FROM table(generate_series(0, 8)) AS g(d);
Why this is better for your case: 1. Automation Friendly: You don't have to calculate the 
LESS THAN
range strings for 
ALTER TABLE ... ADD PARTITION
manually. 2. Safety: It ensures the partitions are created exactly according to your 
date_trunc
logic. 3. Concurrency: As mentioned in the docs, this avoids potential lock conflicts where a heavy load job and a manual 
ALTER TABLE
try to create the same partition at the exact same millisecond when the hour rolls over. Verification After running the command, verify the new partitions exist:
Copy code
sql
SHOW PARTITIONS FROM your_table_name;
Recommendation Since you are concerned about the turn of the hour affecting index building, running this via a cron job or a scheduled task every few hours is a solid "best practice" for high-throughput StarRocks clusters using expression partitioning. References • faq/Others.md • table_design/data_distribution/expression_partitioning.md References • table_design/data_distribution/expression_partitioning.md
Open in Slack
PreviousNext
Powered by