http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> does sr support tiered storage ? I ...
# questions-and-troubleshooting
b
@Rocky does sr support tiered storage ? I want to use s3 express for recent data and s3 standard for old data for some of the table
r
Yes, StarRocks supports tiered storage, but the implementation depends on whether you are using a Shared-nothing or Shared-data cluster. Regarding your specific requirement for S3 Express One Zone and S3 Standard, here is how you can achieve similar results: 1. Tiered Storage in Shared-nothing Clusters (Hot/Cold) In a Shared-nothing architecture, StarRocks allows you to move data between local disks (Hot) and object storage (Cold) automatically based on a time-based policy. * Hot Tier: Local SSD/HDD on the Backend (BE) nodes. * Cold Tier: S3 (Standard or Express). * How it works: You create a 
STORAGE POLICY
and apply it to a table or partition. When data reaches the 
storage_cooldown_time
, StarRocks moves it from local disks to S3. 2. Tiered Storage in Shared-data Clusters (Cache/S3) In Shared-data mode (Cloud-native), all data is persisted in S3. StarRocks uses a Local Data Cache on the Compute Nodes (CN) as the "Hot" tier. * Hot Performance: Data is cached on local NVMe/SSD disks for low-latency access. You can control how long data stays in the cache using 
datacache.partition_duration
. * Cold Tier: The persistent storage in S3 (e.g., S3 Standard). 3. Using S3 Express with S3 Standard AWS S3 Express One Zone (Directory Buckets) and S3 Standard are separate bucket types. Currently, StarRocks does not support an automatic policy to move data between two different S3 buckets (Express to Standard) within the same table. However, you can still use S3 Express in these ways: * As the Primary Storage: Create a 
STORAGE VOLUME
pointing to your S3 Express bucket. In Shared-data mode, this provides the lowest possible latency for all data in that table. * Manual Tiering: If you must use S3 Standard for older data to save costs, you would currently need to manage this at the application level (e.g., by moving older partitions to a different table backed by an S3 Standard storage volume). Recommendation for your Use Case If your goal is performance for recent data and cost-efficiency for old data: 1. Use Shared-data mode with S3 Standard: Enable the Data Cache (
"datacache.enable" = "true"
) and set a 
datacache.partition_duration
(e.g., 
"1 MONTH"
). This keeps recent data on local SSDs (offering performance comparable to or better than S3 Express) while keeping all data in cost-effective S3 Standard. 2. S3 Express for High-IOPS: If you have extremely high-ingestion or metadata-heavy workloads, you can create your 
STORAGE VOLUME
using the S3 Express endpoint. To create a Storage Volume for S3 Express:
Copy code
sql
CREATE STORAGE VOLUME s3_express_vol
TYPE = S3
LOCATIONS = ("<s3://your-directory-bucket--use1-az4--x-s3/>") -- S3 Express bucket format
PROPERTIES
(
    "enabled" = "true",
    "aws.s3.region" = "us-east-1",
    "aws.s3.endpoint" = "<https://s3express-use1-az4.us-east-1.amazonaws.com>"
);
Note: S3 Express support is available as it uses the standard S3-compatible protocol and AWS SDK. References • deployment/shared_data/s3.md • loading/s3.md • data_source/feature-support-data-lake-analytics.md
p
@Rocky Can you see existing storage volume in mysql?
r
Yes, you can see existing storage volumes in StarRocks by using a MySQL client. StarRocks (starting from version 3.1) provides the 
SHOW STORAGE VOLUMES
command specifically for this purpose. When you are connected to StarRocks via a MySQL-compatible client, you can run:
Copy code
sql
SHOW STORAGE VOLUMES;
Key details about this command: * Availability: Supported from StarRocks v3.1 onwards. * Filtering: You can use the 
LIKE
clause to filter for specific volume names:
Copy code
sql
    SHOW STORAGE VOLUMES LIKE 'my_s3%';
* Details: If you need to see more detailed configuration information for a specific storage volume (such as the endpoint, bucket, or cloud provider details), you can use:
Copy code
sql
    DESC STORAGE VOLUME <volume_name>;
Note: This command is specific to StarRocks' architecture (especially for shared-data clusters) and is not a native command in standard MySQL. It only works when the MySQL client is connected to a StarRocks cluster. References • sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md • knowledge/admin_and_operation/shared_data_compaction.md
p
@Rocky if you create s3_express_vol as suggested how do you move tablets from standard s3 bucket to this volume?
r
To move tablets from a standard S3 bucket to a high-performance S3 Express One Zone volume (or any other storage volume), you can use the 
ALTER TABLE
or 
ALTER DATABASE
commands. In StarRocks shared-data clusters, the 
storage_volume
property determines where the physical data (tablets) is stored. When you modify this property for an existing table, StarRocks automatically triggers a background migration to move the data files from the old bucket to the new one. 1. Moving an Entire Table To move all existing data and any future data for a specific table to the new volume:
Copy code
sql
ALTER TABLE <table_name> SET ("storage_volume" = "s3_express_vol");
2. Moving an Entire Database If you want all new tables created in a database to use the new volume, and trigger a migration for existing ones (depending on your version), use:
Copy code
sql
ALTER DATABASE <db_name> SET ("storage_volume" = "s3_express_vol");
Note: This typically sets the default for new tables; for existing tables, it is safer to apply the 
ALTER TABLE
command directly. 3. Moving Specific Partitions If you only want to move specific partitions (e.g., the most recent "hot" data) to S3 Express One Zone while keeping historical data in standard S3:
Copy code
sql
-- Move all existing partitions
ALTER TABLE <table_name> MODIFY PARTITION (*) SET ("storage_volume" = "s3_express_vol");

-- Or move a specific partition by name
ALTER TABLE <table_name> MODIFY PARTITION (p202310) SET ("storage_volume" = "s3_express_vol");
Important Considerations: * Background Migration: After running the 
ALTER
command, StarRocks does not move the data instantly. It starts a background task to copy files between the buckets. * Checking Progress: You can monitor the migration progress using:
Copy code
sql
    SHOW ALTER TABLE COLUMN;
Look for tasks where the 
State
is 
RUNNING
or 
FINISHED
. * Shared-Data Architecture: This feature is exclusive to shared-data clusters. In shared-nothing clusters (where data is on local disks), storage volumes are used differently (mostly for external catalogs or cold data tiering). * S3 Express Endpoint: Ensure your 
s3_express_vol
was created with the correct S3 Express endpoint (e.g., 
<https://s3express-usw2-az1.us-west-2.amazonaws.com>
) and that the IAM role used by StarRocks has permissions for both the source and destination buckets. References • sql-reference/sql-statements/Database/ALTER_DATABASE.md • deployment/shared_data/feature-support-shared-data.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md References • loading/s3.md • deployment/shared_data/s3.md
p
@Rocky in helm chart I still have this these settings under frontend config. Does moving to express one bucket mean I will need to change these? cloud_native_storage_type = S3 aws_s3_path = starrocks-poc/starrocks-cluster-storage aws_s3_region = us-east-1 aws_s3_endpoint = https://s3.us-east-1.amazonaws.com aws_s3_use_aws_sdk_default_behavior = true
r
No, you should not change those settings in your Helm chart if your cluster is already running. In StarRocks shared-data mode, those Helm chart (FE) settings define the cluster's 
builtin_storage_volume
. Changing them on an existing cluster can have critical consequences: 1. Why you should not change them The documentation warns that if you change the original storage path-related configuration items (like 
aws_s3_path
or 
aws_s3_endpoint
) after the cluster is created, the databases and tables created before the change will become read-only, and you will not be able to load new data into them. Essentially, the FE will "lose track" of the metadata's relationship to the physical files in the old bucket. 2. The correct approach: SQL-managed volumes Instead of modifying the global Helm configuration, you should add your S3 Express One Zone bucket as a new, separate storage volume using a SQL command. This allows you to have multiple storage buckets (e.g., standard S3 for cold data and S3 Express for hot data) within the same cluster. Run this command in your MySQL client (ensure you are on StarRocks v3.3.0+ for S3 Express support):
Copy code
sql
CREATE STORAGE VOLUME s3_express_vol
TYPE = S3
LOCATIONS = ("<s3://your-express-bucket--usw2-az1--x-s3/>") -- Use your specific S3 Express bucket name
PROPERTIES (
    "aws.s3.region" = "us-west-2", -- Use the region of your bucket
    "aws.s3.endpoint" = "<https://s3express.us-west-2.amazonaws.com>", -- S3 Express specific endpoint
    "enabled" = "true"
);
3. How to move the data Once the volume is created via SQL, you can move your tables as discussed before:
Copy code
sql
ALTER TABLE <table_name> SET ("storage_volume" = "s3_express_vol");
StarRocks will then handle the background migration from the bucket defined in your Helm chart to the new S3 Express bucket. Summary * Existing Cluster: Keep the Helm settings as they are. Use 
CREATE STORAGE VOLUME
via SQL. * New Cluster (from scratch): You could put the S3 Express details in the Helm chart to make it the default volume, but it is generally more flexible to keep a standard S3 bucket as the "built-in" volume and manage high-performance volumes via SQL. References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • deployment/shared_data/s3.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md
p
@Rocky show example when express one bucket in use1-az4
r
To use an S3 Express One Zone bucket in the 
us-east-1
region specifically within Availability Zone 
use1-az4
, you must follow AWS's strict naming and endpoint requirements. 1. Requirements for S3 Express One Zone * StarRocks Version: You must be on version v3.3.0 or higher. * Bucket Name: Must follow the format 
[base-name]--[azid]--x-s3
. * Endpoint: Must be the zonal endpoint for 
use1-az4
. 2. SQL Example Connect to your StarRocks cluster via a MySQL client and run the following:
Copy code
sql
CREATE STORAGE VOLUME s3_express_use1az4
TYPE = S3
LOCATIONS = ("<s3://my-high-perf-bucket--use1-az4--x-s3/>") 
PROPERTIES (
    "aws.s3.region" = "us-east-1",
    "aws.s3.endpoint" = "<https://s3express-use1-az4.us-east-1.amazonaws.com>",
    "aws.s3.use_aws_sdk_default_behavior" = "true",
    "enabled" = "true"
);
3. Move your data Once the volume is created, you can move a table to this high-performance storage:
Copy code
sql
ALTER TABLE my_table SET ("storage_volume" = "s3_express_use1az4");
Key Details for `use1-az4`: * Bucket Name: If your project name is 
starrocks-data
, your bucket name in the AWS console must be exactly 
starrocks-data--use1-az4--x-s3
. * Regional vs. Zonal Endpoint: While StarRocks can use the regional endpoint, using the zonal endpoint (
<http://s3express-use1-az4.us-east-1.amazonaws.com|s3express-use1-az4.us-east-1.amazonaws.com>
) ensures the lowest possible latency by keeping requests within that specific AZ. * Compute Co-location: For the best performance (single-digit millisecond latency), ensure your StarRocks CN (Compute Nodes) are also deployed in 
use1-az4
. If they are in a different AZ, you may incur cross-AZ data transfer latencies and costs. How to verify After running the 
CREATE
command, verify the volume is active:
Copy code
sql
SHOW STORAGE VOLUMES LIKE 's3_express%';
References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • deployment/shared_data/s3.md
p
@Rocky mysql> ALTER DATABASE test_5299 SET ("storage_volume" = "s3_express_use1az4"); ERROR 1064 (HY000): Getting syntax error at line 1, column 36. Detail message: No viable statement for input 'ALTER DATABASE mosaic_1147_5299 SET ('.
r
The error you're seeing usually occurs for one of two reasons: 1. You are running a Shared-Nothing cluster The 
storage_volume
property and the 
ALTER DATABASE ... SET ("storage_volume" = ...)
command are exclusive to Shared-Data clusters (where data is stored in S3/HDFS and compute is decoupled). * Shared-Nothing: Data is stored on local disks (BE nodes). Databases and tables in this mode do not have a 
storage_volume
property. * Shared-Data: Data is stored in a remote bucket. You can check your cluster mode by running 
SHOW FRONTENDS;
. If the 
IsSharedData
column is 
true
, it is shared-data. If it is 
false
or the column is missing, it is shared-nothing. 2. Version Mismatch or Missing Keyword While the 
ALTER DATABASE
syntax was introduced in version 3.1 for shared-data clusters, some specific sub-versions or client parsers may be picky about the syntax. Try these variations to see if your version requires a slightly different format: Option A: Ensure the database name is quoted (if it has special characters like numbers/underscores)
Copy code
sql
ALTER DATABASE `test_5299` SET ("storage_volume" = "s3_express_use1az4");
Option B: Set it at the Table level (Recommended) The database-level setting primarily sets the default for newly created tables. To move existing data, you must apply the change to the tables themselves. This triggers the background migration of tablets to the new bucket:
Copy code
sql
ALTER TABLE your_table_name SET ("storage_volume" = "s3_express_use1az4");
Option C: Set it for all partitions If the table is partitioned, you might need to target the partitions:
Copy code
sql
ALTER TABLE your_table_name MODIFY PARTITION (*) SET ("storage_volume" = "s3_express_use1az4");
Important Verification Before moving data, ensure the volume was created successfully and is enabled:
Copy code
sql
-- Check if the volume exists and is enabled
SHOW STORAGE VOLUMES;

-- If you are on a Shared-Nothing cluster, this command will return an empty set or an error.
If you are on a Shared-Nothing cluster: You cannot use S3 Express One Zone as the primary storage for your tables. You would instead use S3 for External Catalogs (querying data in S3 without importing it) or Backup/Restore. References • sql-reference/sql-statements/Database/ALTER_DATABASE.md • deployment/shared_data/feature-support-shared-data.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md References • deployment/shared_data/feature-support-shared-data.md • sql-reference/sql-statements/cluster-management/storage_volume/ALTER_STORAGE_VOLUME.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • sql-reference/sql-statements/Database/ALTER_DATABASE.md
Open in Slack
PreviousNext
Powered by