http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> hi rocky, my starrock cluster usin...
# questions-and-troubleshooting
r
@Rocky hi rocky, my starrock cluster using data cache with pvc. but pvc retain policy remain. what happen if i delete data cache volume when cluster stop?
r
Hi there! Rocky here to help. If you delete the PVC/volume used for Data Cache while the StarRocks cluster is stopped, here is what will happen: *1. No Data Loss* You will not lose any persistent data. In a StarRocks shared-data architecture (or when using Data Cache for external tables), the "source of truth" is your remote storage (like S3, GCS, or HDFS). The Data Cache on the PVC is only a local copy used to accelerate queries. *2. Cold Start Performance Hit* When you restart the cluster, StarRocks will see that the cache directory is empty. * Initial Queries: The first time you run queries after the restart, they will be significantly slower. StarRocks must fetch the data from the remote storage over the network. * Automatic Re-caching: As those queries run, StarRocks will automatically pull the data from remote storage and populate the new volumes. Over time, as the cache warms up, performance will return to normal. *3. Metadata and Cache Consistency* StarRocks is designed to handle missing cache files gracefully. If the cache is gone, it simply treats it as a "cache miss" and goes to the backend storage. You don't need to perform any manual re-indexing or configuration changes; the system handles the empty volume automatically upon startup. Summary of Impact: Aspect | Result Data Integrity | Safe. No primary data is lost. Query Speed | Slow initially. High latency until the cache is rebuilt ("Cold Start"). Cluster Health | Normal. The BE/CN nodes will start up fine and create new cache files. Rocky's Tip: If you are doing this to clear up disk space or fix a corrupted disk, it is a perfectly safe operation. Just be prepared for a period of higher I/O and network usage while the cluster "warms up" its cache again! References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • deployment/shared_data/feature-support-shared-data.md
r
@Rocky how can i give delete option if i using operator?
r
If you are using the StarRocks Kubernetes Operator to manage your cluster and want to enable or "give" the DELETE option, it typically refers to one of three things: granting SQL 
DELETE
privileges to a user, configuring the Operator's scale-down behavior, or managing resource deletion (like PVCs). Here is how you can handle each scenario: 1. Granting SQL 
DELETE
Privileges If you want a specific database user to have the ability to delete data from tables (SQL 
DELETE
statement), you must perform this via SQL after the cluster is deployed by the Operator. The Operator itself does not currently manage internal database user privileges via the YAML CRD. Connect to your StarRocks cluster (via the FE service) and run:
Copy code
sql
-- Grant DELETE privilege on a specific table
GRANT DELETE ON TABLE <database_name>.<table_name> TO USER '<user_name>'@'%';

-- Or grant it at the database level
GRANT DELETE ON ALL TABLES IN DATABASE <database_name> TO USER '<user_name>'@'%';
_Note: Only users with the 
db_admin
role or 
DELETE
privilege can execute delete operations. If you are using Primary Key tables, 
DELETE
is fully supported._ 2. Enabling "Delete" (Scale-Down) in the Operator If you are using the 
autoScalingPolicy
in your 
StarRocksCluster
YAML and want to control whether the Operator is allowed to delete pods (scale down) when load is low: In your 
StarRocksCluster
spec, look at the 
behavior
section:
Copy code
yaml
starRocksCnSpec:
  autoScalingPolicy:
    hpaPolicy:
      behavior:
        scaleDown:
          selectPolicy: Min # Allows scaling down (deleting pods)
          # Use "Disabled" if you want to prevent the operator from deleting pods automatically
If you set 
selectPolicy: Disabled
, the Operator will not delete pods even if the metrics suggest scaling down. 3. Deleting the Cluster vs. Retaining Data If you are looking for the "option" to delete the cluster while keeping your data, or ensuring data is deleted when the cluster is removed: * To delete the cluster: Run 
kubectl delete starrockscluster <cluster_name>
. * PVC Retention: By default, the Operator manages 
PersistentVolumeClaims
. If you want to ensure the data (PVs) is deleted when the cluster is deleted, ensure your 
StorageClass
has a 
reclaimPolicy: Delete
. If you want to keep the data, use 
reclaimPolicy: Retain
. 4. Granting the 
OPERATE
Privilege If you mean the OPERATE system-level privilege (which allows a user to perform node-level operations like 
ALTER SYSTEM
), use:
Copy code
sql
GRANT OPERATE ON SYSTEM TO USER '<user_name>'@'%';
Summary for your "Operator" context: * To delete data: Use SQL 
GRANT DELETE
. * To delete pods (scale-in): Check 
autoScalingPolicy.behavior.scaleDown
in your YAML. * To delete the cluster: Use 
kubectl delete
. If you were referring to a specific "delete" flag in a command-line tool or a different type of operator, please let me know!
r
@Rocky i mean delete option for pvc(data cache) when cluster stop. how can i delete pvc when i stop cluster or delete cr with starrock operator? give me some option.
r
For the StarRocks Operator, by default, PersistentVolumeClaims (PVCs) are not deleted when you stop the cluster (scale down) or delete the 
StarRocksCluster
Custom Resource (CR). This is standard Kubernetes behavior for 
StatefulSets
to prevent accidental data loss. If you want to ensure that the PVCs used for Data Cache (typically used in shared-data mode on CN or BE nodes) are deleted when the cluster is removed or stopped, here are your options: Option 1: Use 
emptyDir
for Data Cache (Recommended for Ephemeral Cache) Since Data Cache is reconstructible from remote storage, you can use 
emptyDir
instead of a PVC. This ensures that the storage is tied to the Pod's lifecycle—when the Pod is deleted or the cluster stops, the cache is automatically wiped. In your 
StarRocksCluster
YAML, you can define a custom volume for the cache path:
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  starRocksCnSpec: # or starRocksBeSpec
    replicas: 3
    # Use emptyDir for the data cache directory
    storageSpec:
      - name: data-cache
        mountPath: /opt/starrocks/cn/storage # Default storage path
        emptyDir: {}
Option 2: StatefulSet PVC Retention Policy (K8s 1.27+ Stable) If your Kubernetes version is 1.23 or higher, you can use the 
persistentVolumeClaimRetentionPolicy
within the 
StatefulSet
. This allows you to specify that PVCs should be deleted when the cluster is scaled down or deleted. While the StarRocks Operator primarily manages storage through 
storageSpec
, you can check if your Operator version supports passing the 
persistentVolumeClaimRetentionPolicy
field. It usually looks like this:
Copy code
yaml
spec:
  starRocksBeSpec:
    # This policy controls what happens to PVCs
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Delete
      whenScaled: Delete
Note: Support for this specific field depends on the version of the StarRocks Operator you are running (v1.8.0+ generally supports more StatefulSet-native fields). Option 3: Manual Cleanup via Labels If you prefer to keep using PVCs for performance reasons (e.g., using specific SSD storage classes) but want a quick way to delete them when you delete the cluster, you can use labels. 1. Delete the Cluster:
Copy code
bash
   kubectl delete starrockscluster <cluster_name>
2. Delete the associated PVCs: The StarRocks Operator labels all PVCs with the cluster name. You can delete them all at once:
Copy code
bash
   kubectl delete pvc -l "<http://starrocks.com/cluster-name=<cluster_name>|starrocks.com/cluster-name=<cluster_name>>"
Option 4: Change Reclaim Policy of the StorageClass You can set the 
reclaimPolicy
of your 
StorageClass
to 
Delete
. However, this only deletes the physical volume (PV) when the PVC is deleted; it does not trigger the deletion of the PVC itself when the CR is removed. You still have to delete the PVC manually or via the retention policy mentioned in Option 2. Summary * For pure cache (reconstructible): Use Option 1 (
emptyDir
). It is the cleanest way to ensure no leftover storage costs. * For persistent cache across pod restarts but delete on cluster removal: Use Option 2 if supported, otherwise use Option 3. References • quick_start/helm.md • deployment/sr_operator.md • deployment/shared_data/s3.md
r
@Rocky if i using empty dir , node can get volume pressure if there are many cn pod in same k8s node. how can give volume upper bound for cn node?, and what happend if many pod write a lot in same noe using empty dir?
r
Managing StarRocks CN (Compute Node) pods with 
emptyDir
in Kubernetes requires careful configuration to avoid node-level disk pressure and pod evictions. 1. How to set a volume upper bound for CN nodes When using 
emptyDir
for storage (which is common for logs, query spill-to-disk, or data caching), you can limit the volume size in two ways: A. Kubernetes Level: 
ephemeral-storage
Limits Since 
emptyDir
volumes are backed by the node's local storage, they are part of the pod's "ephemeral storage." You can set a limit in your 
StarRocksCluster
YAML under the `starRocksCnSpec`:
Copy code
yaml
starRocksCnSpec:
  requests:
    cpu: 8
    memory: 32Gi
    ephemeral-storage: "50Gi"
  limits:
    cpu: 8
    memory: 32Gi
    ephemeral-storage: "100Gi" # K8s will evict the pod if it exceeds this limit
B. StarRocks Level: Data Cache Limits If your CN nodes use local storage for Data Cache (common in shared-data mode), you should explicitly limit how much disk space the cache can consume. By default, StarRocks might use up to 80% of the available disk. In your 
cn.conf
(via 
configMap
or 
commonSpec.config
), set: • `starlet_star_cache_disk_size_percent`: The percentage of disk capacity the Data Cache can use (default is 
80
). • `datacache_disk_size`: An absolute byte value for the cache (e.g., 
21474836480
for 20GB). Example configuration in the Operator:
Copy code
yaml
starRocksCnSpec:
  config: |
    starlet_star_cache_disk_size_percent = 50
    # or
    datacache_disk_size = 53687091200
2. What happens if many pods write a lot to the same node? If multiple CN pods on the same K8s node write heavily to 
emptyDir
(logs, cache, or spilling), the following will occur: * Node Disk Pressure: Kubernetes monitors the disk usage of the underlying filesystem (usually 
/var/lib/kubelet
). If usage exceeds the eviction threshold (typically 85-90%), the node will enter a 
DiskPressure
state. * Pod Eviction: To reclaim space, the Kubelet will start evicting pods. Pods that do not have 
ephemeral-storage
requests/limits or those that are exceeding their limits are the first candidates for eviction. * I/O Contention: All CN pods on that node share the same physical disk bandwidth. High write volume from one pod (e.g., during a massive query spill) will increase IO-wait and significantly slow down query performance for all other pods on that same node. * Query Failures: If the disk fills up completely before K8s can evict a pod, StarRocks may fail to write necessary logs or spill data, leading to 
Internal Error
or 
Disk Full
errors during query execution. Recommendation For production environments with multiple CN pods per node: 1. Use Persistent Volumes (PVs): Instead of 
emptyDir
, use a 
storageSpec
with a PVC (using local SSD or high-performance cloud storage) to provide isolated and guaranteed storage for each pod. 2. Set Resources: Always define 
ephemeral-storage
limits to ensure K8s can manage node stability. 3. Monitor: Keep an eye on the 
starlet_cache
and 
spill
directories to ensure they don't grow unexpectedly. References • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md • sql-reference/System_variable.md • administration/management/FE_configuration.md
Open in Slack
PreviousNext
Powered by