http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> What is the recommended approach fo...
# questions-and-troubleshooting
r
@Rocky What is the recommended approach for managing users, roles, and privileges in the StarRocks FE? How can we ensure that user accounts, roles, and their associated privileges remain intact even if FE pods are deleted and the underlying EBS volumes are also removed?
r
Managing users, roles, and privileges in StarRocks follows a Role-Based Access Control (RBAC) model. To ensure these remain intact even if FE pods and their underlying storage (EBS) are completely removed, you should move away from relying solely on the FE's local metadata storage. Here is the recommended approach for management and persistence: 1. Recommended Management Approach (RBAC) The best practice is to manage permissions at the Role level rather than the User level. * Use System Roles: Assign system-defined roles like 
db_admin
, 
user_admin
, or 
cluster_admin
to your platform administrators. * Service-Specific Roles: Create custom roles for different business lines (e.g., 
analyst_role
, 
developer_role
) and grant specific database/table privileges to these roles. * Assign Roles to Users: Grant the roles to users. This makes it easier to manage permissions at scale and audit access. 2. Ensuring Persistence After Storage Loss In StarRocks, metadata (including users, roles, and privileges) is stored in the FE's local directory (
meta_dir
) using Berkeley DB Java Edition (BDBJE). If the EBS volumes are deleted, this internal metadata is lost. To prevent this, use the following strategies: *A. External Authorization (Apache Ranger) - Recommended for Enterprise* By integrating StarRocks with Apache Ranger, you externalize the authorization logic. * How it works: StarRocks will check permissions against Ranger policies rather than its internal metadata. * Resilience: Even if your StarRocks cluster is completely wiped and rebuilt, as long as you point the new cluster to your existing Ranger service, all roles and privileges remain intact because they are stored in the Ranger database (usually backed by an external RDS/MySQL). B. External Authentication (LDAP/Azure AD) + Group Mapping Instead of manually creating user accounts in StarRocks, use Security Integrations (LDAP, OAuth2, or Azure AD). * How it works: Users authenticate via your corporate directory. * Resilience: You can map external groups (e.g., an AD group "Data_Analysts") to StarRocks roles. If the cluster is recreated, you only need to recreate the mapping; the user accounts themselves do not need to be manually re-registered. C. Infrastructure as Code (IaC) / SQL Scripts Treat your RBAC configuration as code. * How it works: Maintain all 
CREATE USER
, 
CREATE ROLE
, and 
GRANT
statements in a version-controlled Git repository. * Resilience: Use a CI/CD pipeline or an initialization script to apply these SQL statements to the cluster. This allows you to restore the entire privilege structure in seconds on a fresh cluster. D. Periodic Metadata Backup While StarRocks supports data backups to S3 via the 
BACKUP
command, this primarily covers table data. * For Metadata: You should set up a periodic job to back up the FE 
meta_dir
(specifically the image files and BDBJE logs) to an external object store like S3. * Recovery: If EBS is lost, you can download the backup to the new FE's 
meta_dir
and start the FE with the 
bdbje_reset_election_group = true
configuration to recover the catalog and users. Summary of Recommendations Scenario | Solution Complete Storage Loss | Use Apache Ranger for privileges and LDAP for users. Configuration Management | Use SQL scripts in Git to recreate RBAC structures. K8s Best Practice | Ensure your FE Pods use 
PersistentVolumeClaims
with a Retain Reclaim Policy to prevent accidental EBS deletion. References • deployment/shared_data/s3.md • deployment/shared_data/minio.md
r
@Rocky I’m running StarRocks on Kubernetes and using Vault in my platform. What is the recommended way to manage users, roles, and privileges in StarRocks FE? Specifically: 1. Can Vault be used to manage StarRocks users/roles, or should it only store credentials? 2. What’s the best practice for automating RBAC? Looking for production best practices. Thanks!
r
Managing users and privileges in a production StarRocks environment on Kubernetes involves a multi-layered approach. Since StarRocks is MySQL-compatible, you can leverage existing MySQL-based tools, but there are StarRocks-native ways to optimize this. Here is the recommended architecture for managing users, roles, and privileges using Vault and Kubernetes. 1. The Role of HashiCorp Vault In a StarRocks ecosystem, Vault should primarily store and manage credentials, rather than managing the internal RBAC logic (roles/privileges). * Credential Storage: Store the 
root
password and application-specific user passwords in Vault. * Dynamic Secrets: Since StarRocks is MySQL-compatible, you can often use the Vault MySQL Secrets Engine to generate dynamic, time-bound credentials for applications connecting to StarRocks. * Kubernetes Integration: Use the Vault Agent Sidecar Injector to mount these credentials as files or environment variables into your application pods. For the StarRocks FE itself, use Vault to manage the 
root
password secret that the StarRocks Operator or Helm chart references. 2. Best Practices for Authentication (Identity) For production, avoid creating "native" users manually in StarRocks (
CREATE USER ... IDENTIFIED BY 'password'
). Instead, use Security Integration. * LDAP/AD Integration: Connect StarRocks to your corporate identity provider (LDAP or Active Directory). This ensures that when an employee leaves the company, their access is revoked automatically. * Security Integration Object: Use the 
CREATE SECURITY INTEGRATION
command. This allows StarRocks to delegate authentication to LDAP without needing to store user passwords locally.
Copy code
sql
    CREATE SECURITY INTEGRATION ldap_int
    PROPERTIES (
        "type" = "authentication_ldap_simple",
        "authentication_ldap_simple_server_host" = "<http://ldap.example.com|ldap.example.com>",
        "authentication_ldap_simple_bind_base_dn" = "ou=users,dc=example,dc=com",
        ...
    );
3. Best Practices for Authorization (RBAC) While authentication is external (LDAP), Authorization (privileges) should be managed via StarRocks Roles. * Group Mapping (The Pro Way): Use a Group Provider. This allows you to map LDAP groups directly to StarRocks roles. _ _Example:* Anyone in the LDAP group 
data_science_team
is automatically assigned the StarRocks role 
transformer_role
. * This eliminates the need to manually run 
GRANT
commands for every new user. * RBAC Hierarchy: * Define functional roles (e.g., 
read_only_role
, 
db_admin_role
, 
pbi_reporter_role
). * Grant privileges to these roles, not to individuals. * Enable 
activate_all_roles_on_login = TRUE
so users don't have to manually execute 
SET ROLE
. 4. Automating RBAC (Infrastructure as Code) To manage roles and privileges at scale in Kubernetes, do not use manual SQL scripts. * Terraform: Use the Terraform StarRocks Provider. This is the standard for production automation. You can define your roles, databases, and group mappings in HCL code. * Vault provides the provider with the administrative credentials. * Terraform ensures the state of your roles matches your Git repository (GitOps). * Kubernetes Operator: While the StarRocks Operator handles the lifecycle of the nodes (FE/BE), it is generally not used for fine-grained RBAC. Use Terraform or a specialized 
Job
that runs SQL migrations (like Flyway or Liquibase) to manage the schema and privileges. Summary Checklist for Production: 1. Secrets: Use Vault to store the 
root
password; inject into K8s via Secrets or Sidecar. 2. Auth: Implement Security Integration with LDAP/AD. 3. Privileges: Use Group Providers to map external groups to internal StarRocks roles. 4. Automation: Use Terraform to manage roles and privileges as code.
5. Audit: Enable Audit Logging in the FE configuration (
fe.conf
) to track who performed what action, which is a requirement for most Vault-managed environments. References • administration/user_privs/authorization/User_privilege.md • administration/user_privs/authorization/user_privs.md • administration/user_privs/authorization/privilege_item.md
r
@Rocky How to enable audit logging?
r
In StarRocks, audit logging is enabled by default to record queries in a local file. You can also configure the AuditLoader plugin to store these logs in a StarRocks table for easier querying and analysis. Here are the two ways to manage audit logging: 1. Default File-based Audit Logging By default, StarRocks records audit logs in the 
fe/log/fe.audit.log
file on each Frontend (FE) node. * Key Configuration Parameters (in 
fe.conf
): * `audit_log_modules`: Specifies which modules to log. Default is 
slow_query, query
. Since v3.0, you can also add 
connection
. * `audit_log_dir`: The directory where audit logs are stored. * `audit_log_roll_num`: Number of history log files to keep (default: 90). * `audit_log_roll_interval`: Rotation frequency (default: 
DAY
). * `audit_log_json_format`: Set to 
true
to log in JSON format (default: 
false
). 2. Table-based Audit Logging (via AuditLoader Plugin) To query audit logs using SQL, you can use the AuditLoader plugin to load logs into a StarRocks table. Step 1: Create the Audit Database and Table First, create a destination for the logs. It is recommended to use a Duplicate Key table with dynamic partitions:
Copy code
sql
CREATE DATABASE starrocks_audit_db__;

CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ (
  `queryId` VARCHAR(64),
  `timestamp` DATETIME NOT NULL,
  `queryType` VARCHAR(12),
  `clientIp` VARCHAR(32),
  `user` VARCHAR(64),
  `authorizedUser` VARCHAR(64),
  `resourceGroup` VARCHAR(64),
  `catalog` VARCHAR(32),
  `db` VARCHAR(96),
  `state` VARCHAR(8),
  `errorCode` VARCHAR(512),
  `queryTime` BIGINT,
  `scanBytes` BIGINT,
  `scanRows` BIGINT,
  `returnRows` BIGINT,
  `cpuCostNs` BIGINT,
  `memCostBytes` BIGINT,
  `stmtId` INT,
  `isQuery` TINYINT,
  `feIp` VARCHAR(128),
  `stmt` VARCHAR(1048576),
  `digest` VARCHAR(32),
  `planCpuCosts` DOUBLE,
  `planMemCosts` DOUBLE,
  `pendingTimeMs` BIGINT
) ENGINE = OLAP
DUPLICATE KEY (`queryId`, `timestamp`, `queryType`)
PARTITION BY date_trunc('day', `timestamp`)
PROPERTIES (
  "replication_num" = "3",
  "partition_live_number" = "30"
);
Step 2: Configure AuditLoader 1. Download the AuditLoader package. 2. Unzip it and modify 
plugin.conf
with your cluster details: * `frontend_host_port`: 
127.0.0.1:8030
(recommended) * `database`: 
starrocks_audit_db__
* `table`: 
starrocks_audit_tbl__
* `user`: Your username (requires 
LOAD_PRIV
) * `password`: Your password 3. Zip the files back into 
auditloader.zip
and distribute it to all FE nodes at the same absolute path. Step 3: Install the Plugin Run the following SQL command in StarRocks:
Copy code
sql
INSTALL PLUGIN FROM "/absolute/path/to/auditloader.zip";
Step 4: Verification Verify the installation by running:
Copy code
sql
SHOW PLUGINS;
Once installed, the plugin will periodically (default every 60s) load new entries from the local audit log file into the 
starrocks_audit_tbl__
table. You can then query it like any other table:
Copy code
sql
SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ 
WHERE queryTime > 1000; -- Find queries slower than 1s
Note: If you need to update the configuration, you must first run 
UNINSTALL PLUGIN AuditLoader;
, update the zip file, and then reinstall it. References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md
r
@Rocky Is their a way we can ship fe logs to AWSOpensearch?
r
Yes, you can ship StarRocks FE logs to AWS OpenSearch. StarRocks doesn't have a native "push" integration for OpenSearch built into its core configuration, so the standard and recommended way to do this is by using a log agent (like Filebeat or Fluent Bit) running on your FE nodes. Here is the recommended approach to set this up: 1. Enable JSON Formatting for FE Logs By default, StarRocks logs are in a text format that requires complex parsing. To make ingestion into OpenSearch much easier, you should enable JSON formatting for audit and profile logs in your 
fe/conf/fe.conf
file:
Copy code
properties
# Enable JSON format for audit logs
audit_log_json_format = true

# Profile logs are JSON by default in newer versions, 
# but you can ensure they are enabled:
enable_profile_log = true
Note: The main 
fe.log
and 
fe.warn.log
are standard Log4j text logs. You will need a multiline parser for these to handle Java stack traces. 2. Use a Log Forwarder (Agent) Install a log forwarder on each FE node to tail the log files and send them to your AWS OpenSearch endpoint. Option A: Filebeat (Recommended) Filebeat is lightweight and has a dedicated output for OpenSearch/Elasticsearch. * Input: Point it to the StarRocks log directory (usually 
${STARROCKS_HOME}/fe/log/
). * Process: Use the 
decode_json_fields
processor for 
fe.audit.log
. * Output: Point to your AWS OpenSearch domain endpoint. Option B: Fluent Bit Fluent Bit is very popular in Kubernetes/Cloud environments. * Tail Plugin: Monitors 
fe.log
, 
fe.warn.log
, and 
fe.audit.log
. * Parser: Use a JSON parser for the audit logs and a multiline regex parser for the system logs. * AWS Output: Use the 
es
output plugin (OpenSearch compatible) with the appropriate AWS IAM permissions. 3. Log Locations to Monitor Ensure your agent is watching these specific files in your StarRocks FE log directory: * 
fe.audit.log
: Critical for tracking SQL queries, user activity, and performance (now in JSON). * 
fe.log
: The main system log for cluster state and general info. * 
fe.warn.log
: Essential for troubleshooting errors and performance bottlenecks. * 
fe.out
: Standard output/error, useful for catching JVM-level crashes or startup failures. 4. Directing to AWS OpenSearch (Security) If your OpenSearch cluster is inside a VPC: * Ensure the FE nodes have the necessary Security Group rules to communicate with the OpenSearch endpoint. * If using IAM-based access control, the log agent will need access to AWS credentials (e.g., via an IAM Instance Profile) to sign requests. Summary of Benefits * Centralization: View logs from all FE nodes in one dashboard. * Alerting: Set up monitors in OpenSearch to alert you on specific keywords like "FATAL" or "ERROR". * Analytics: Use OpenSearch Dashboards to visualize query latencies and slow queries based on the JSON audit logs. References • administration/management/FE_configuration.md
r
@Rocky Why v*ault should primarily store and manage credentials*, rather than managing the internal RBAC logic (roles/privileges).?
r
Hi, I'm Rocky! That is a great architectural question. While HashiCorp Vault is an incredibly powerful tool, there are several key reasons why it is best practice to use it for credential management (Secret Management) while leaving RBAC logic (Privileges) to the database engine like StarRocks (or an authorization-specific service like Apache Ranger). Here is why this separation of concerns is generally preferred: 1. Separation of Concerns * Vault's Role (Identity & Secrets): Vault is designed to answer the question: "Is this person/service who they say they are, and can I give them a valid password or token?" It excels at the lifecycle of secrets—encryption, rotation, and time-to-live (TTL). * StarRocks' Role (Authorization): StarRocks is designed to answer the question: "Now that I know who this user is, do they have permission to SELECT from Table A or DROP Table B?" The database has the full context of its own internal objects (catalogs, databases, tables, views). 2. Granularity and Complexity Database privileges are extremely granular. In StarRocks, you might have hundreds of tables with different permissions (
SELECT
, 
INSERT
, 
ALTER
) across multiple catalogs. * If you tried to manage this in Vault, you would have to map every single database object to a Vault path or policy. * This creates a "double-configuration" nightmare where every time you create a new table in StarRocks, you have to update a policy in Vault. 3. Performance and Latency StarRocks is a high-performance OLAP engine built for sub-second queries. * Internal RBAC: Checking a privilege internally is a memory-speed operation. * External RBAC via Vault: If StarRocks had to call a Vault API for every single query to check if a user has 
SELECT
access, it would introduce significant network latency and overhead, potentially becoming a bottleneck for your entire data platform. 4. Reliability and Availability If your RBAC logic lives entirely inside Vault: * Single Point of Failure: If Vault is down or the network is slow, no queries can run because the database cannot "authorize" them. * By keeping RBAC internal to StarRocks (or synced to its FE nodes), the cluster can continue to process queries even if the external secret manager is momentarily unreachable. 5. Idiomatic Management StarRocks provides standard SQL commands (
GRANT
, 
REVOKE
, 
CREATE ROLE
) to manage access. Data engineers and DBAs are familiar with this workflow. Managing SQL-level permissions through Vault's HCL policies or API is often non-intuitive for the teams actually managing the data. The "Golden Path" Integration The most common way to use them together is: 1. Vault generates Dynamic Credentials for a user (e.g., a temporary username/password for StarRocks). 2. The user connects to StarRocks with those credentials. 3. StarRocks looks at its internal RBAC tables (or an external authorizer like Apache Ranger) to see what that specific username is allowed to do. This gives you the security of Vault's secret rotation with the performance and granularity of StarRocks' native authorization. For more details on how StarRocks handles its internal privileges, you can check out the StarRocks RBAC documentation. References • administration/user_privs/authorization/User_privilege.md • best_practices/authentication_authorization.md • administration/user_privs/authentication/security_integration.md • administration/user_privs/authorization/user_privs.md
r
@Rocky If Vault generates Dynamic Credentials for a user (e.g., a temporary username/password for StarRocks). how Starrocks will now that user has access to a specific table or not?
r
When HashiCorp Vault generates Dynamic Credentials for StarRocks, StarRocks tracks permissions through its native RBAC (Role-Based Access Control) system. Because StarRocks is MySQL-compatible, it integrates with Vault using the Vault MySQL Database Secrets Engine. Here is the step-by-step breakdown of how StarRocks "knows" what that dynamic user can access: 1. Vault Executes Grant Statements When you configure a "Role" in HashiCorp Vault for StarRocks, you define 
creation_statements
. These are standard StarRocks SQL commands. For example:
Copy code
sql
-- Vault Role Configuration (Creation Statements)
CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';
GRANT SELECT ON db1.table1 TO '{{name}}'@'%';
When a user requests credentials, Vault connects to StarRocks (using a pre-configured administrative account) and executes these exact statements. 2. StarRocks Stores the Privileges Locally StarRocks is not "aware" that the user was created by Vault. To StarRocks, this is simply a new user created via standard SQL. * Metadata Storage: The StarRocks Frontend (FE) stores this user and the granted privileges in its internal catalog (the FE image and edit logs). * Consistency: This information is synchronized across all FE nodes in the cluster, ensuring any FE the user connects to knows their permissions. 3. Permission Enforcement at Query Time When the dynamic user logs into StarRocks and runs a query: 1. Authentication: StarRocks verifies the username and password against its internal user list (created by Vault in Step 1). 2. Authorization: Before executing the query, the StarRocks FE checks its internal metadata to see if that specific username has the 
SELECT
(or other) privilege on the requested table. 3. Result: If the 
GRANT
statement executed by Vault included access to that table, the query proceeds. 4. Revocation (Lease Expiry) When the Vault lease expires or is revoked: 1. Vault connects to StarRocks again. 2. Vault executes the 
revocation_statements
(usually 
DROP USER '{{name}}'@'%';
). 3. StarRocks removes the user and all associated privileges from its metadata. Any subsequent attempt to use those credentials will result in an "Access Denied" error. Summary StarRocks knows the permissions because Vault explicitly tells StarRocks what the permissions are (via 
GRANT
statements) at the moment the user is created. StarRocks then enforces those permissions using its own internal engine just like any other manually created user. Rocky's Tip: Ensure the "Root" user you give to Vault has the 
user_admin
or 
grant_admin
system-level permissions in StarRocks so it can successfully create and manage these temporary users! References • sql-reference/System_variable.md • introduction/what_is_starrocks.md • faq/resource_isolation_faq.md • data_source/External_table.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • sql-reference/sql-functions/Window_function.md • faq/Others.md • sql-reference/information_schema/engines.md
r
@Rocky What is the use of LOG_CONSOLE in fe/cn conf?
r
In StarRocks, 
LOG_CONSOLE
 (often used as an environment variable) and its corresponding configuration parameters are used to control whether system logs are output to the standard output (stdout) or standard error (stderr), in addition to being written to log files. This is primarily used in containerized environments (such as Docker or Kubernetes) to allow log collection agents (like 
kubectl logs
, Fluentd, or Grafana Loki) to capture the logs directly from the container's output stream. 1. For Frontend (FE) In the FE, the equivalent configuration in 
fe.conf
is: * 
sys_log_to_console
: * Type: Boolean (`true`/`false`) * Default: 
false
* Description: When set to 
true
, FE system logs (which normally go to 
fe.log
) are also printed to the console. 2. For Compute Node (CN) and Backend (BE) While 
LOG_CONSOLE
is not always a direct entry in the 
cn.conf
file, it is a widely used environment variable in the StarRocks startup scripts (
start_cn.sh
/ 
start_be.sh
) and Helm charts. * Usage: 
export LOG_CONSOLE=1
or 
export LOG_CONSOLE=true
* Effect: * It directs the process to output logs to the console. * In many deployment scenarios (like the StarRocks Kubernetes Operator), setting this ensures that the logs are visible when you run 
kubectl logs <pod_name>
. * Without this, you would typically need to 
exec
into the pod and look at the files in the 
/opt/starrocks/cn/log/
directory. Summary of Benefits * Observability: Makes logs easily accessible via standard container tools. * Debugging: Allows you to see real-time startup logs without needing to access the file system. * Storage: In some ephemeral container setups, printing to the console is preferred over storing large log files on a small container root disk. Note: If you are running on physical machines or VMs and using a traditional log rotation service, you generally keep this disabled (
false
) to avoid cluttering the system's 
stdout
and potentially duplicating log data. References • administration/management/FE_configuration.md • deployment/sr_operator.md • administration/management/BE_configuration.md
r
@Rocky What is the difference between Back up and restore data and Cluster Snapshot?
r
In StarRocks, Back up and restore data and Cluster Snapshot are two different mechanisms designed for different cluster architectures and use cases. Here is a breakdown of the key differences: 1. Supported Architecture * Back up and restore data: Primarily designed for Shared-nothing (classic) clusters. Shared-data clusters currently do not support the standard 
BACKUP
and 
RESTORE
commands. * Cluster Snapshot: Exclusively available for Shared-data clusters (from v3.4.2 onwards). 2. Scope of Content * Back up and restore data: Offers granular control. You can choose to back up specific databases, individual tables, or even just certain partitions. It also supports metadata for external catalogs, Materialized Views, and UDFs. * Cluster Snapshot: Captures the entire cluster state. It is a "full-cluster" snapshot that includes all metadata (FE state) and data (CN state), along with users, privileges, and loading tasks. It is not intended for restoring a single table. 3. Purpose and Use Case * Back up and restore data: Best for data migration (e.g., moving a database from one cluster to another) or protecting specific high-value tables. * Cluster Snapshot: Designed for disaster recovery (DR). In a shared-data architecture, data is already in object storage, but metadata is local to the FEs. Cluster Snapshot ensures the metadata is also safely stored in object storage, allowing you to recover the entire cluster if the original FE nodes are lost. 4. Automation and Lifecycle * Back up and restore data: This is a manual, point-in-time operation. You manually run the 
BACKUP
command, and StarRocks creates a snapshot in your repository (S3, HDFS, etc.). You can keep multiple historical backups. * Cluster Snapshot: The system automatically maintains a snapshot that closely follows the latest cluster state. It typically keeps only the most recent snapshot available, dropping historical ones as new ones are created. Comparison Summary Feature | Back up and restore data | Cluster Snapshot Cluster Type | Shared-nothing | Shared-data Granularity | Database / Table / Partition | Entire Cluster Storage | S3, HDFS, GCS, MinIO | Object Storage (S3/Azure/GCS/OSS) Primary Use | Migration / Specific table recovery | Full Cluster Disaster Recovery User Access | Manual commands (`BACKUP`/`RESTORE`) | Automated system maintenance Includes Users/Privs | No | Yes Which one should you use? * Use Back up and restore if you are on a standard shared-nothing cluster and want to move data or save specific tables. * Use Cluster Snapshot if you are on a shared-data cluster and want a "safety net" to recover the entire environment in case of a catastrophic failure. References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md
r
@Rocky I am seeing these WARN in FE pod logs while doing the DR activity.
Copy code
2026-02-25 10:54:57.177+08:00 WARN (replayer|108) [ClusterSnapshotMgr.replayLog():429] Invalid Cluster Snapshot Log Type UPDATE_SNAPSHOT_JOB
r
Hello! I'm Rocky. The 
WARN
message you're seeing during your Disaster Recovery (DR) activity is related to the Cluster Snapshot feature (a shared-data cluster feature introduced in StarRocks v3.4.2). What this warning means The message 
Invalid Cluster Snapshot Log Type UPDATE_SNAPSHOT_JOB
indicates that during the metadata replay process, the FE encountered an edit log entry meant to update the status of a snapshot job (e.g., moving a job from 
SNAPSHOTING
to 
FINISHED
), but the replayer logic in that specific version of the 
ClusterSnapshotMgr
did not recognize this specific log type. Key Points: 1. Severity: This is a WARNING, not an error. It generally does not block the metadata recovery or the cluster restoration process. 2. Impact: The primary impact is usually limited to the metadata state of the snapshot jobs themselves. Specifically, the view 
information_schema.cluster_snapshot_jobs
might show an inconsistent state (like a job stuck in 
SNAPSHOTING
) because the update log was skipped during replay. 3. DR Integrity: The actual snapshot data (the metadata image and data files stored in your object storage) is usually unaffected. As long as the snapshot was successfully created and uploaded to the storage volume before the DR event, the restoration should proceed. Recommended Actions: * Verify Restoration: Check if the cluster has successfully restored the metadata and if your data is accessible. You can run 
SELECT * FROM information_schema.cluster_snapshots;
to see if the snapshot you are restoring from is correctly recognized. * Check Job Status: Run 
SELECT * FROM information_schema.cluster_snapshot_jobs;
to see if there are any failed jobs, but keep in mind the warning means some of these statuses might be out of date. * Version Check: Since Cluster Snapshot is currently a Beta feature, ensure you are using the latest maintenance release of the v3.4.x branch, as refinements to the log replay logic are frequently included in these updates. If your cluster successfully completes the recovery and data is intact, you can safely ignore this warning. If the restoration fails or stalls, please check the FE logs for any subsequent 
ERROR
messages. References • administration/cluster_snapshot.md • administration/management/FE_configuration.md
r
@Rocky Is their any prometheus metrics which gives status of whether automatic cluster snapshot working properly or not?
r
There are currently no dedicated Prometheus metrics specifically for monitoring the status of "Automatic Cluster Snapshots" (introduced in v3.4.2). However, you can monitor the status and health of automatic cluster snapshots using the following methods: 1. Monitor via SQL View (Recommended) The primary way to check if snapshots are working correctly is by querying the 
information_schema.cluster_snapshot_jobs
view. You can look for jobs in an 
ERROR
state.
Copy code
sql
SELECT * FROM information_schema.cluster_snapshot_jobs 
WHERE state = 'ERROR';
Field | Description state | The state of the job. Valid values: 
INITIALIZING
, 
SNAPSHOTING
, 
FINISHED
, 
EXPIRED
, 
DELETED
, and 
ERROR
. error_message | Provides the specific error reason if the job failed. You can also verify the latest successful snapshot:
Copy code
sql
SELECT * FROM information_schema.cluster_snapshots 
ORDER BY created_time DESC LIMIT 1;
2. Monitor Metadata Checkpoint Health (Prometheus) Automated cluster snapshots are triggered every time a metadata checkpoint is completed. Therefore, if the FE metadata checkpoint process fails, snapshots will not be created. You can monitor the following Prometheus metric to detect checkpoint failures: * Metric: 
starrocks_fe_meta_log_count
* Description: The number of edit logs waiting for a checkpoint. * Alert Rule: If this value exceeds 100,000 (default checkpoint threshold is 50,000), it usually indicates that the checkpoint process has failed, which will stop new snapshots from being generated. 3. Monitor FE Logs For detailed troubleshooting, you can check the 
fe.log
on the Leader FE node for the following patterns: * Successful Checkpoint: 
checkpoint finished save image.xxxx
* Snapshot Start: Look for log entries related to 
automated_cluster_snapshot
or 
ClusterSnapshotJob
. Summary for Alerting If you need to set up automated alerting in Prometheus/Grafana: 1. Indirect Alert: Use 
starrocks_fe_meta_log_count > 100000
to catch checkpoint failures. 2. Direct Alert: Use a SQL Exporter to periodically run 
SELECT count(*) FROM information_schema.cluster_snapshot_jobs WHERE state = 'ERROR'
and expose that count as a metric to Prometheus. References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md
r
@Rocky I want to create databases in starrocks cluster but not by the root user. What privileges should I give to a role which can be used be the user in that role to create database and create tbales etc.
r
To create databases and tables in StarRocks without using the 
root
user, you can either use the built-in 
db_admin
 role or create a custom role with specific privileges. Option 1: Use the built-in 
db_admin
role StarRocks provides a system-defined role called 
db_admin
which is designed specifically for database management. It has all the necessary privileges to manage data but cannot manage users or cluster nodes. You can grant this role to your user:
Copy code
sql
GRANT db_admin TO USER 'your_user'@'%';
Option 2: Create a Custom Role If you want to create a more restricted role that specifically allows creating databases and tables, you need to grant privileges at two levels: the Catalog level (to create the database) and the Database level (to create tables within them). 1. Create the role:
Copy code
sql
    CREATE ROLE database_creator;
2. Grant privilege to create databases: In StarRocks, the permission to create a database is managed at the Catalog level (usually the 
default_catalog
).
Copy code
sql
    GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE database_creator;
3. Grant privilege to create tables: To allow the user to create tables inside databases, you can grant the 
CREATE TABLE
privilege. You can grant this for all current and future databases in a catalog:
Copy code
sql
    GRANT CREATE TABLE ON ALL DATABASES IN CATALOG default_catalog TO ROLE database_creator;
4. Grant the role to your user:
Copy code
sql
    GRANT database_creator TO USER 'your_user'@'%';
Summary of Key Privileges Task | Privilege | Level Create a Database | 
CREATE DATABASE
| 
CATALOG
Create a Table | 
CREATE TABLE
| 
DATABASE
Manage existing tables | 
ALL
or 
SELECT, INSERT, etc.
| 
TABLE
or 
DATABASE
Note: If your user needs to perform other actions like creating Views or Materialized Views, you should also grant 
CREATE VIEW
or 
CREATE MATERIALIZED VIEW
at the 
DATABASE
level. For a user to fully manage the databases they create (e.g., dropping them or altering them), you might consider granting 
ALL
on the catalog or specific databases. References • administration/user_privs/authorization/privilege_item.md • administration/user_privs/authorization/user_privs.md • sql-reference/sql-statements/Database/CREATE_DATABASE.md • administration/user_privs/authorization/built_in_roles.md
r
@Rocky What are the recommended permissions and type of users should we created and their privileges.
r
For managing a StarRocks cluster effectively, we recommend using Role-Based Access Control (RBAC). This involves creating specific roles based on job functions and assigning those roles to users. Here are the recommended user types, roles, and their associated privileges: 1. Cluster Maintainers (Super Users/Admins) These users are responsible for the health of the entire cluster, including node management and global security. * Recommended Roles: Use the system-defined roles. * `cluster_admin`: Node management (adding/deleting FE/BE). * `user_admin`: User and privilege management. * `db_admin`: Management of all databases and objects. * Best Practice: Assign these to a platform user but set 
db_admin
and 
user_admin
as default, requiring manual activation of 
cluster_admin
only when performing node operations.
Copy code
sql
    GRANT db_admin, user_admin, cluster_admin TO USER platform_mgr;
    ALTER USER platform_mgr DEFAULT ROLE db_admin, user_admin;
2. Department or LOB Administrators These users manage a specific set of databases belonging to a business unit (Line of Business). * Privileges: Full control over their specific databases and the ability to grant permissions to their team members. * Example:
Copy code
sql
    CREATE ROLE dept_a_admin;
    GRANT SELECT, ALTER, INSERT, UPDATE, DELETE, DROP ON ALL TABLES IN DATABASE db_dept_a TO ROLE dept_a_admin WITH GRANT OPTION;
    GRANT dept_a_admin TO USER manager_a;
3. Data Analysts / Power Users (Read-Write) Users who need to load data, create temporary tables, and perform complex analysis. * Privileges: 
SELECT
on production tables, plus 
CREATE TABLE
and 
INSERT
on specific development/staging databases. * Example:
Copy code
sql
    CREATE ROLE data_analyst;
    GRANT USAGE ON ALL CATALOGS TO ROLE data_analyst;
    GRANT SELECT ON ALL TABLES IN DATABASE prod_db TO ROLE data_analyst;
    GRANT ALL ON ALL TABLES IN DATABASE staging_db TO ROLE data_analyst;
4. Business Viewers (Read-Only) Standard users who only need to query data or view dashboards. * Privileges: Global or database-level read access. * Example:
Copy code
sql
    CREATE ROLE read_only;
    GRANT USAGE ON ALL CATALOGS TO ROLE read_only;
    GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_only;
    GRANT SELECT ON ALL VIEWS IN ALL DATABASES TO ROLE read_only;
    -- Also include UDFs if they are used in queries
    GRANT USAGE ON ALL FUNCTIONS IN ALL DATABASES TO ROLE read_only;
5. Service Accounts (ETL / Applications) Accounts used by automated tools (like Flink, DataX, or your application backend). * Privileges: Strictly limited to what the tool needs. For ETL, this usually means 
INSERT
and 
SELECT
. For applications, typically just 
SELECT
. * Example (ETL Load Role):
Copy code
sql
    CREATE ROLE etl_role;
    GRANT INSERT, SELECT ON ALL TABLES IN DATABASE target_db TO ROLE etl_role;
Summary of Best Practices 1. Least Privilege: Never grant more permissions than necessary. Start with 
USAGE
and 
SELECT
. 2. Use Roles, Not Direct Grants: Always grant privileges to a Role, then assign the Role to a User. This makes it much easier to manage permissions as your team grows. 3. The 
public
Role: StarRocks has a built-in 
public
role. Every user is a member by default. Use this for datasets that everyone in the company should be able to see (e.g., a 
common_dim
database).
Copy code
sql
    GRANT SELECT ON ALL TABLES IN DATABASE common_dim TO ROLE public;
4. Password Policy: For all manual users, ensure complex passwords are set during creation:
Copy code
sql
    CREATE USER 'username'@'%' IDENTIFIED BY 'Complex_Password_123';
5. Default Roles: If a user has multiple roles, use 
ALTER USER ... DEFAULT ROLE ...
so they don't have to manually execute 
SET ROLE
every time they log in. References
r
@Rocky What are the recommended prometheus metrics I should watch for and put alert . What are their recommended threshold be. I have 3 FE and 2 CN pods on t3a.xlarge EC2 nodes running
r
For your StarRocks cluster with 3 FEs and 2 CNs running on t3a.xlarge instances (4 vCPU, 16GB RAM), here are the recommended Prometheus metrics and alert thresholds. Since t3a.xlarge is a burstable instance type, you should be particularly cautious about CPU exhaustion and memory limits. 1. Service Availability (Critical) These alerts tell you if your cluster is healthy or if a node has crashed. Metric / Alert | PromQL / Logic | Recommended Threshold | Why? FE Node Status | 
starrocks_fe_status
| Alert if 
0
| 1 = Alive, 0 = Dead. With 3 FEs, you can survive 1 failure, but you should fix it immediately. CN Node Status | 
starrocks_be_status
| Alert if 
0
| CNs are stateless but provide the compute. With only 2 CNs, losing one cuts your capacity by 50%. FE Alive Count | 
count(starrocks_fe_status == 1)
| 
< 3
| Triggers if any of your 3 FEs go down. CN Alive Count | 
count(starrocks_be_status == 1)
| 
< 2
| Triggers if any of your 2 CNs go down. ────────── 2. Resource Utilization (Performance) Given your 16GB RAM and 4 vCPU per node, these thresholds help prevent OOM (Out of Memory) or CPU bottlenecks. Metric / Alert | Recommended Threshold | Why? FE JVM Heap | 
> 80%
| FE stores metadata in memory. If this hits 100%, FE will OOM. For 16GB RAM, your heap is likely 8GB. Alert at 6.4GB. CN Mem Usage | 
> 90%
| CNs use memory for query execution and caching. If it hits the limit (defined by 
mem_limit
), it will start killing queries. CN CPU Idle | 
< 10%
(for 30s) | Since 
t3a
instances use CPU credits, sustained 90%+ usage will quickly exhaust your credits, leading to severe throttling. Disk Available | 
< 20%
| Even though CNs are stateless, they need disk for logs and intermediate query "spilling." ────────── 3. Metadata & System Health Metric / Alert | Recommended Threshold | Why? FE Meta Log | 
> 100,000
| Indicates FE checkpointing is failing. This can lead to very long FE restart times or metadata corruption. FE Thread Count | 
> 3,000
| High thread counts (default limit 4096) usually indicate too many concurrent 
UNION ALL
or complex queries. Failed Txns | 
> 5%
(rate over 5m) | Alert if data loading (Stream Load/Routine Load) is failing frequently. ────────── 4. Query Performance Metric / Alert | Recommended Threshold | Why? Query Latency (P99) | Set based on baseline | If your usual P99 is 200ms and it jumps to 2s, something is wrong (likely resource contention on your 4 vCPUs). Query Error Rate | 
> 1%
| Catch SQL syntax errors or resource-related query cancellations. Special Advice for 
t3a.xlarge
 1. CPU Credits: Monitor the 
aws_ec2_cpucredit_balance
(if using CloudWatch exporter). If it drops to 0, your StarRocks performance will collapse. 2. FE Heap Size: Ensure your 
fe.conf
has 
-Xmx8g
(or similar). Since your node has 16GB, don't set it too high, as the OS and other processes need room. 3. CN Memory: Ensure 
mem_limit
in 
cn.conf
is set to roughly 
80-90%
of the total 16GB. How to get these metrics: All these metrics are exposed by StarRocks at: * FE: 
http://<fe_ip>:<fe_http_port>/metrics
* CN: 
http://<cn_ip>:<be_http_port>/metrics
References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/alert.md
r
@Rocky How to grant EXPLORE and 
OPERATE
PERMISSIONS to a role?
r
r
@Rocky Starrocks > GRANT EXPORT -> ON SYSTEM -> TO ROLE sr_admin_role; ERROR 1064 (HY000): Getting analyzing error. Detail message: Cannot grant or revoke EXPORT on 'SYSTEM' type object.
r
The error occurs because the 
EXPORT
privilege cannot be granted at the SYSTEM level in StarRocks. In the StarRocks RBAC (Role-Based Access Control) model, the 
EXPORT
privilege is associated with Table or Database objects, not the System object. How to fix it Depending on the scope you want to grant, use one of the following commands: 1. Grant global export privileges (all tables in all databases): If you want the role to be able to export data from any table in the cluster:
Copy code
sql
GRANT EXPORT ON ALL TABLES IN ALL DATABASES TO ROLE sr_admin_role;
2. Grant export privileges on a specific database:
Copy code
sql
GRANT EXPORT ON ALL TABLES IN DATABASE <db_name> TO ROLE sr_admin_role;
3. Grant export privileges on a specific table:
Copy code
sql
GRANT EXPORT ON TABLE <table_name> TO ROLE sr_admin_role;
Supported privileges on SYSTEM For your reference, the privileges that can be granted on 
SYSTEM
include: * 
CREATE RESOURCE GROUP
* 
CREATE RESOURCE
* 
CREATE EXTERNAL CATALOG
* 
REPOSITORY
(Used for backup/restore) * 
BLACKLIST
* 
FILE
* 
OPERATE
* 
CREATE STORAGE VOLUME
* 
SECURITY
(Used for managing users/roles) References • sql-reference/sql-statements/account-management/GRANT.md • unloading/Export.md • administration/user_privs/authorization/User_privilege.md
r
@Rocky How to expose fe logs to Opensearch? I have fluentbit running as deamon set and runs as pod on the same node where starrocks fe pod spins up. On giving this :
Copy code
[INPUT]
  Name             tail
  Tag              kube.*
  Path             /var/log/containers/*.log,/opt/starrocks/fe/log/*.log
  Parser           cri
  DB               /var/log/flb_kube.db
  Mem_Buf_Limit    100MB
  Skip_Long_Lines  On
  Refresh_Interval 10
I am seeing
Copy code
[2026/02/26 13:07:06] [ warn] [input:tail:tail.0] error scanning path: /opt/starrocks/fe/log/*.log
[2026/02/26 13:07:16] [error] [input:tail:tail.0] read error, check permissions: /opt/starrocks/fe/log/*.log
What is the recommended way to get fe pod logs to opensearch via fluentbit?
r
To expose StarRocks FE logs to OpenSearch using a Fluent Bit DaemonSet, you need to address the fact that Fluent Bit (running as a DaemonSet on the host) cannot see files inside another pod's container filesystem unless those files are explicitly shared via a host volume or streamed to 
stdout
. The error you are seeing (
error scanning path: /opt/starrocks/fe/log/*.log
) occurs because Fluent Bit is looking for that path on the host node, where it doesn't exist. Here are the recommended ways to get FE logs to OpenSearch: Option 1: The "Kubernetes Native" Way (Recommended) Kubernetes best practice is to have containers log to 
stdout/stderr
. Fluent Bit is already configured to collect these from 
/var/log/containers/*.log
. However, StarRocks FE writes to files (like 
fe.log
) by default. To bridge this, you can add a sidecar container to your FE Pod that tails the log files to its own standard output. Update your FE Pod/StatefulSet spec:
Copy code
yaml
      containers:
      - name: starrocks-fe
        # ... your existing FE config ...
        volumeMounts:
        - name: fe-log
          mountPath: /opt/starrocks/fe/log
      
      # Sidecar to tail fe.log to stdout
      - name: fe-log-tail
        image: busybox:1.36
        command: ["/bin/sh", "-c", "tail -F /opt/starrocks/fe/log/fe.log"]
        volumeMounts:
        - name: fe-log
          mountPath: /opt/starrocks/fe/log
      
      volumes:
      - name: fe-log
        emptyDir: {} # Or your existing log PVC
Why this works: * The sidecar 
fe-log-tail
outputs the log content to its 
stdout
. * Kubernetes captures this in 
/var/log/containers/
. _ Your existing Fluent Bit configuration (
Path /var/log/containers/_.log
) will automatically pick it up. ────────── Option 2: Shared HostPath (Direct File Access) If you prefer Fluent Bit to read the files directly from the disk (e.g., to handle multiline logs or specific audit logs like 
fe.audit.log
), you must mount a 
hostPath
volume into both the StarRocks FE pod and the Fluent Bit DaemonSet. 1. In StarRocks FE Spec: Mount a host path for logs.
Copy code
yaml
    volumeMounts:
    • name: fe-logs-host
      mountPath: /opt/starrocks/fe/log
    volumes:
    • name: fe-logs-host
      hostPath:
        path: /var/log/starrocks/fe
        type: DirectoryOrCreate
2. In Fluent Bit DaemonSet Spec: Mount the same host path so Fluent Bit can see it.
Copy code
yaml
    volumeMounts:
    • name: starrocks-logs
      mountPath: /mnt/starrocks/fe/log
      readOnly: true
    volumes:
    • name: starrocks-logs
      hostPath:
        path: /var/log/starrocks/fe
3. Update Fluent Bit Config: Change the path to match where you mounted it in Fluent Bit:
Copy code
[INPUT]
      Name             tail
      Tag              starrocks.fe.*
      Path             /mnt/starrocks/fe/log/*.log
      Parser           cri
      ...
────────── Option 3: Modify Log4j to Log to Console StarRocks FE uses 
log4j
. Advanced users can modify the 
log4j.properties
(or 
log4j2.xml
in newer versions) located in the FE 
conf
directory to add a 
ConsoleAppender
. If you are using the StarRocks Operator or Helm Chart, you can often inject a custom log4j configuration via a ConfigMap. This allows 
fe.log
content to go directly to the pod's 
stdout
, removing the need for sidecars or host volumes. Summary Checklist * Permissions: If you use Option 2, ensure the Fluent Bit user has read permissions for 
/var/log/starrocks/fe
on the host. * Multiline Parsing: StarRocks logs (especially Java stack traces in 
fe.log
) are multiline. Ensure your Fluent Bit 
Parser
or 
Filter
is configured with 
multiline.parser
to avoid splitting stack traces into multiple OpenSearch documents.
* Audit Logs: If you need 
fe.audit.log
for security/auditing, Option 1 (using multiple sidecars or a single sidecar tailing multiple files) or Option 2 is usually easiest since audit logs are distinct from system logs. References • administration/management/logs.md • administration/management/FE_configuration.md • quick_start/helm.md
r
@Rocky What is the recommended way to calculate CN CPU Idle to show in the grafana. Which metrics should be used and what should be the promql for that?
r
To calculate and display the CPU Idle percentage for Compute Nodes (CN) in Grafana using StarRocks metrics, you should use the built-in system metrics exported by the processing nodes. In StarRocks, both BE (Backend) and CN (Compute Node) share the same metric names and are prefixed with 
starrocks_be_
because they use the same underlying engine. Recommended Metric and PromQL The most straightforward way to show CPU Idle is using the pre-calculated 
cpu_idle
metric. 1. CPU Idle Percentage To show the percentage of time the CPU is idle: * Metric Name: 
starrocks_be_cpu_idle
* PromQL:
Copy code
promql
    starrocks_be_cpu_idle{job="starrocks_cn"}
_(Note: Replace 
job="starrocks_cn"
with the specific job name or label you use in your Prometheus configuration to distinguish CN nodes from BE nodes.)_ 2. CPU Usage (Calculated) If you prefer to show CPU Usage instead of Idle, you can subtract the idle value from 100: * PromQL:
Copy code
promql
    100 - starrocks_be_cpu_idle{job="starrocks_cn"}
_Alternatively, StarRocks also exports 
starrocks_be_cpu_util
which represents the usage rate directly._ Configuration Details * Metric Type: Gauge/Average (0-100). * Source: These metrics are gathered from 
/proc/stat
on the host where the CN process is running and are exposed via the CN's HTTP port (default 
8040
) at the 
/metrics
endpoint. * Grafana Visualization: * Panel Type: Gauge, Stat, or Time series. * Unit: Percent (0-100). * Thresholds: It is recommended to alert if the Idle rate is lower than 10% for a sustained period (e.g., 30-60 seconds), as this indicates a CPU bottleneck. How to distinguish CN from BE Since both export metrics with the 
starrocks_be_
prefix, ensure your Prometheus 
scrape_configs
includes a label to identify the node type:
Copy code
yaml
scrape_configs:
  - job_name: 'starrocks_cn'
    static_configs:
      - targets: ['cn_host1:8040', 'cn_host2:8040']
        labels:
          group: 'cn'
Then your PromQL becomes: 
starrocks_be_cpu_idle{group="cn"}
. References • administration/management/monitoring/metrics.md
r
@Rocky There is no such metrics coming in prometheus. starrocks_be_cpu{mode="Idle"} is present though. How to use this one?
r
Hi there! I'm Rocky. It is normal to see 
starrocks_be_cpu
with the 
mode="Idle"
(or 
mode="idle"
) label. In StarRocks, this metric tracks the cumulative CPU time for various modes. To calculate the CPU Utilization percentage for your BE nodes using the idle metric, you can use the following PromQL formula in Prometheus/Grafana:
Copy code
promql
(1 - sum(rate(starrocks_be_cpu{mode="idle"}[5m])) by (instance) / sum(rate(starrocks_be_cpu[5m])) by (instance)) * 100
How this works: * 
sum(rate(starrocks_be_cpu[5m]))
: This calculates the total CPU time rate across all modes (user, system, idle, etc.). * 
sum(rate(starrocks_be_cpu{mode="idle"}[5m]))
: This calculates the rate of CPU time specifically spent being idle. * The Division: Dividing idle time by total time gives you the "Idle Ratio." * 
1 - ...
: Subtracting the idle ratio from 1 gives you the "Usage Ratio." * ` 100`*: Converts the ratio to a percentage. Things to note: 1. Case Sensitivity: If your Prometheus specifically shows the label as 
mode="Idle"
(capitalized), make sure to match that casing in your query. 2. Granularity: The 
[5m]
(5-minute range) is a recommendation to smooth out spikes. You can adjust this to 
[1m]
for more real-time sensitivity. 3. Alerting: A common alert rule is to trigger if the Idle rate is lower than 10% (meaning utilization is > 90%) for a sustained period. If you are looking for other specific CPU metrics like "User" or "System" time, they are usually labels under that same 
starrocks_be_cpu
metric name. You can explore them by running 
starrocks_be_cpu
in your Prometheus expression browser. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • administration/management/monitoring/alert.md
r
@Rocky Is their a way I can automate the creation of snapshot storage volume while deploying the FE and CN using the helm chart?
Copy code
CREATE STORAGE VOLUME backup_volume 
TYPE = S3 LOCATIONS = ("<s3://starrocks-backup-test>") 
PROPERTIES (     
"enabled" = "true",     
"aws.s3.region" = "us-east-1",     
"aws.s3.use_aws_sdk_default_behavior" = "true",     
"aws.s3.use_instance_profile" = "false", 
"aws.s3.endpoint"="<https://s3.us-east-1.amazonaws.com>"
r
Automating the creation of a 
STORAGE VOLUME
(such as for snapshots or shared data) during a Helm deployment can be achieved in two ways depending on whether you want a built-in default volume or a custom-named volume. Method 1: Automating the "Built-in" Storage Volume (Recommended for Shared-Data) If you are deploying a shared-data cluster and want this storage volume to be the primary one used by the system, you can use the 
enable_load_volume_from_conf
feature. This tells the FE to automatically create a volume named 
builtin_storage_volume
using the properties defined in your configuration. In your 
values.yaml
, you would add these to the FE configuration:
Copy code
yaml
starrocks:
  starrocksFESpec:
    configMapInfo:
      configMapName: fe-config
      resolveKey: fe.conf
  # Define the configmap in your chart or via extraObjects
extraObjects:
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: fe-config
    data:
      fe.conf: |
        run_mode = shared_data
        enable_load_volume_from_conf = true
        aws_s3_path = starrocks-backup-test
        aws_s3_region = us-east-1
        aws_s3_endpoint = <https://s3.us-east-1.amazonaws.com>
        aws_s3_use_aws_sdk_default_behavior = true
        aws_s3_use_instance_profile = false
Method 2: Automating Custom SQL via a Kubernetes Job (Best for Snapshots) If you need to create a specific volume named 
backup_volume
(as shown in your example) after the cluster is up, the most reliable way is to use a Kubernetes Job with a Helm 
post-install
hook. You can add a template to your Helm chart (e.g., 
templates/init-storage-volume.yaml
) or deploy it alongside your Helm install:
Copy code
yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: create-storage-volume-job
  annotations:
    "<http://helm.sh/hook|helm.sh/hook>": post-install
    "<http://helm.sh/hook-weight|helm.sh/hook-weight>": "5"
    "<http://helm.sh/hook-delete-policy|helm.sh/hook-delete-policy>": hook-succeeded
spec:
  template:
    spec:
      containers:
      - name: mysql-client
        image: mysql:8.0 # Or any image with a mysql client
        command: ["/bin/sh", "-c"]
        args:
          - |
            # Wait for FE to be ready
            until mysql -h kube-starrocks-fe-service -P 9030 -u root -e "SELECT 1"; do
              echo "Waiting for FE..."
              sleep 5
            done
            # Execute the create volume command
            mysql -h kube-starrocks-fe-service -P 9030 -u root -e '
            CREATE STORAGE VOLUME backup_volume 
            TYPE = S3 LOCATIONS = ("<s3://starrocks-backup-test>") 
            PROPERTIES (     
              "enabled" = "true",     
              "aws.s3.region" = "us-east-1",     
              "aws.s3.use_aws_sdk_default_behavior" = "true",     
              "aws.s3.use_instance_profile" = "false", 
              "aws.s3.endpoint"="<https://s3.us-east-1.amazonaws.com>"
            );'
      restartPolicy: OnFailure
Summary of Key Points: * Built-in Volume: Use the FE config 
enable_load_volume_from_conf = true
. This is native to StarRocks and requires no external scripts, but the volume is always named 
builtin_storage_volume
. * Custom Volume: Use a Kubernetes 
Job
. This is the standard way to automate post-deployment SQL (like creating users, volumes, or databases) in a Kubernetes environment. * Security: If you have enabled 
initPassword
in the Helm chart, ensure the Job uses the password from the Kubernetes secret instead of an empty password. References • quick_start/shared-nothing.md • administration/management/resource_management/Blacklist.md • sql-reference/System_variable.md • sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md
2 Views
Open in Slack
PreviousNext
Powered by