http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> it seems fe leader node is not lett...
# questions-and-troubleshooting
a
@Rocky it seems fe leader node is not letting follower nodes take on read operations because CPU on two follower fe nodes is barely utilized, any reason why?
👍 1
r
The reason your Follower FE nodes are underutilized is likely that StarRocks does not automatically distribute incoming query connections from the Leader FE to Follower/Observer nodes. In a StarRocks cluster: 1. Connection Handling: The FE node that receives the client connection is responsible for parsing, planning, and scheduling the query. If all your clients/applications are configured to connect only to the Leader FE's IP address, the Leader will perform all that work, leaving the Followers' CPUs idle. 2. Write vs. Read: While write requests (like 
INSERT
or 
LOAD
) are internally routed to the Leader FE to ensure metadata consistency, read queries are executed on whichever FE the client is currently connected to. 3. No Internal Load Balancer: The Leader FE does not act as a proxy or load balancer for read queries received on its own port. How to distribute the load to Follower FEs: To utilize your Followers for read operations, you need to distribute the client connections across all FE nodes using one of the following methods: * External Load Balancer: Deploy a load balancer (such as ProxySQL, Nginx, HAProxy, or a cloud-native LB) in front of your FE nodes. Configure it to distribute connections among the Leader and Followers (and Observers, if any). * JDBC Load Balancing: If you are using a Java application, you can use the MySQL JDBC driver's built-in load balancing by providing a comma-separated list of FE IPs in the connection string:
Copy code
sql
    jdbc:mysql:<loadbalance://fe_ip1:9030,fe_ip2:9030,fe_ip3:9030/db_name>
* Client-Side Distribution: Manually configure different application instances to point to different FE IP addresses. Summary: To see CPU utilization on your Followers, you must ensure that some of your client connections are physically landing on those Follower nodes. Check your 
SHOW FRONTENDS;
output to ensure all nodes are 
Alive: true
and then balance your connection strings. References • administration/management/resource_management/Load_balance.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • faq/Deploy_faq.md
a
@Rocky what requests and limits do you recommend for starrocksFeProxySpec on the helmchart and how many replicas?
r
For the 
starrocksFeProxySpec
 in the StarRocks Helm chart, this component acts as a lightweight Nginx-based proxy. Its primary purpose is to provide a single entry point for MySQL clients and to handle external Stream Load requests by correctly routing traffic between FEs and BEs (especially when BEs are not directly reachable from outside the Kubernetes cluster). Here are the recommendations for its configuration: 1. Replicas * Recommendation: At least 2 replicas (3 for production high availability). * Reason: Since the FE Proxy is the entry point for your traffic, having only one replica creates a single point of failure. If you are using a 
LoadBalancer
service type, it will distribute traffic across these proxy pods. 2. Resource Requests and Limits Because the FE Proxy is an Nginx-based service, it is significantly more lightweight than the FE or BE nodes. It does not perform heavy computation or data storage. * CPU: * Request: 
200m
to 
500m
* Limit: 
1
(Nginx can scale well with CPU, but rarely needs more than 1 core unless you have extremely high concurrency). * Memory: * Request: 
256Mi
to 
512Mi
* Limit: 
512Mi
to 
1Gi
(Memory usage is primarily for connection buffers and is generally very stable). Example Configuration Snippet You can add this to your `values.yaml`:
Copy code
yaml
starrocks:
  starrocksFeProxySpec:
    enabled: true
    replicas: 2
    service:
      type: LoadBalancer
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: "1"
        memory: 512Mi
Key Considerations: * Stream Load: If you perform very high-throughput data loading via Stream Load, you may want to increase the CPU limits to handle the SSL termination (if configured) and the high volume of HTTP requests. * Scaling: If you notice the proxy pods hitting their CPU limits during peak load, you should scale the replicas horizontally rather than just increasing the limits, as this provides better redundancy and leverages Kubernetes' load balancing. * FE/BE Resources: Note that the actual database nodes (FE/BE) require much higher resources. For a production environment, FEs typically start at 8 cores / 16GB and BEs at 16 cores / 64GB+. References • quick_start/helm.md • deployment/plan_cluster.md • administration/management/FE_configuration.md • deployment/environment_configurations.md • administration/management/BE_configuration.md • administration/management/resource_management/resource_group.md
a
@Rocky why does added feproxyspec keep getting deleted after I enabled it on existing cluster in helmchart?
r
Based on your description, the most likely reason the 
feProxySpec
(or 
starrocksFeProxySpec
) keeps getting deleted is a version mismatch between your Helm chart and the StarRocks Operator (or its CRDs). Here is why this happens and how to fix it: 1. The Cause: Outdated CRDs The 
feProxySpec
field was introduced in later versions of the StarRocks Operator (typically v1.8.0+). If you are enabling this feature on an existing cluster: * The Problem: Your Kubernetes cluster likely still has the older version of the Custom Resource Definition (CRD) for 
StarRocksCluster
. * The Result: When you run 
helm upgrade
with the new proxy configuration, Helm sends the updated spec to Kubernetes. However, because the old CRD schema does not define 
feProxySpec
, the Kubernetes API server treats it as an "unknown field" and automatically strips it out (deletes it) during the save process. 2. How to Fix It You need to update the StarRocks Operator and, crucially, the CRDs to the latest version. Step A: Update the CRDs manually Helm does not automatically update CRDs during a 
helm upgrade
. You must apply them manually from the StarRocks Operator repository:
Copy code
bash
kubectl apply -f <https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml>
Step B: Update the StarRocks Operator Ensure your operator is running a version that supports the proxy (v1.8.1 or higher is recommended). If you used Helm to install the operator:
Copy code
bash
helm repo update
helm upgrade <operator-release-name> starrocks/starrocks-operator
Step C: Verify the Helm Value Structure Ensure you are using the correct key in your 
values.yaml
. For most official Helm charts, the structure should look like this (under the 
starrocks
root):
Copy code
yaml
starrocks:
  starrocksFeProxySpec:
    enabled: true
    # For external access, use LoadBalancer or NodePort
    service:
      type: LoadBalancer
Note: Some versions of the chart use 
starrocksFeProxySpec
in 
values.yaml
which maps to 
feProxySpec
in the actual Kubernetes object. 3. Other Potential Issues * Manual Edits vs. Helm: If you are using 
kubectl edit
to add the field while a Helm release or a GitOps tool (like ArgoCD) is managing the cluster, the tool will automatically revert (delete) your manual changes to match what is defined in the chart's 
values.yaml
. Always update the 
values.yaml
and run 
helm upgrade
. * Indentation: Kubernetes YAML is sensitive to indentation. Ensure 
starrocksFeProxySpec
is at the same level as 
starrocksFESpec
and 
starrocksBeSpec
. Summary Checklist 1. Apply the latest CRD using the 
kubectl apply
command mentioned above. 2. Restart the Operator pod to ensure it recognizes the new schema. 3. Re-run your 
helm upgrade
 with the proxy enabled in your 
values.yaml
. References • quick_start/helm.md • deployment/sr_operator.md • administration/management/resource_management/Load_balance.md • deployment/deploy_manually.md
a
@Rocky Starrocks operator keeps killing my feproxyspec. Here are the logs
StarRocksClusterReconciler.feProxyController.feProxyController delete deployment from kubernetes
r
Hello! I'm Rocky. It sounds like the StarRocks Operator's reconciliation loop is determining that the FE Proxy component should not exist based on your current 
StarRocksCluster
(SRC) configuration. When you see the log 
StarRocksClusterReconciler.feProxyController.feProxyController delete deployment from kubernetes
, it means the operator is trying to synchronize the cluster's state with your YAML definition, and it believes the FE Proxy is not requested or is misconfigured. Here are the most common reasons why this happens: 1. Incorrect Field Name in the Spec The StarRocks Operator is case-sensitive and expects specific field names. If you are using 
feProxySpec
or 
feproxyspec
, the operator will ignore it and assume no proxy is required, subsequently deleting any existing proxy deployment. * Correct Field Name: 
starRocksFeProxySpec
2. Missing 
enabled: true
flag In many versions of the StarRocks Operator (and especially when using Helm), the FE Proxy must be explicitly enabled within its spec. If the 
enabled
field is missing or set to 
false
, the controller will delete the deployment. Example of a correct configuration:
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  # ... other specs (fe, be, etc.)
  starRocksFeProxySpec:  # Ensure this prefix and casing are correct
    enabled: true        # This is often required
    replicas: 1
    image: starrocks/fe-proxy:latest # Use an appropriate image
    service:
      type: LoadBalancer # Common use case for Stream Load
3. Manual Deployment Conflict If you created the FE Proxy deployment manually (e.g., via 
kubectl create deployment
) using names or labels that overlap with what the operator manages, the 
feProxyController
may see it as an "unmanaged" or "extra" resource and delete it to maintain the state defined in your 
StarRocksCluster
custom resource. * Solution: Always define the FE Proxy within the 
StarRocksCluster
resource rather than as a standalone deployment. 4. FE Component Dependency The FE Proxy controller typically waits for the FE nodes to be ready. If your FE nodes are crashing or in a 
Pending
state, the operator might occasionally cycle the dependent components like the FE Proxy. Check the status of your FE pods:
Copy code
bash
kubectl get pods -l "<http://starrocks.com/cluster-name=<your-cluster-name>,starrocks.com/component=fe|starrocks.com/cluster-name=<your-cluster-name>,starrocks.com/component=fe>"
Next Steps: * Check your 
StarRocksCluster
YAML for the 
starRocksFeProxySpec
field. * Verify that 
enabled: true
is present. * Run 
kubectl describe starrockscluster <name>
to see if there are any validation errors or events explaining why the sub-component is being removed. If you can share the 
starRocksFeProxySpec
section of your YAML, I can help you spot any syntax issues! References • quick_start/helm.md • administration/management/FE_configuration.md • administration/management/resource_management/Load_balance.md
a
@Rocky here is my starrocksFeProxySpec, can yousee any issues 
starrocksFeProxySpec:
# specify the FE proxy deployment or not.
 
enabled: true
replicas: 1
imagePullPolicy: IfNotPresent
# default nginx:1.24.0
 
image:
repository: "starrocks/fe-proxy"
tag: "latest"
resources:
requests:
cpu: 1
memory: 2Gi
limits:
cpu: 1
memory: 2Gi
# set the resolver for nginx server, default kube-dns.kube-system.svc.cluster.local
 
resolver: ""
service:
# the FE proxy service type, only supported ClusterIP, NodePort, LoadBalancer
 
# default ClusterIP
 
type: ClusterIP
# the loadBalancerIP for static ip config when the type=LoadBalancer and loadbalancerIp is not empty.
 
loadbalancerIP: ""
# add annotations for external FE proxy service.
 
annotations: {}
# Add labels for external FE proxy service. The operator may add its own default labels.
 
labels: {}
# Config the service port for FE proxy service.
 
# To assign a specific port or nodePort to a service, you should specify them by the corresponding name or
 
# containerPort in the service configuration. If both containerPort and name are specified, containerPort takes precedence.
 
# For FE proxy, the port name can be http-port, and its default container port is 8080.
 
ports: []
# e.g., specify a dedicated node port for FE proxy service by containerPort.
 
# - nodePort: 30080 # The range of valid ports is 30000-32767
 
#   containerPort: 8080 # The port on the container to expose.
 
# Specify the source IP ranges for the load balancer when the type=LoadBalancer.
 
loadBalancerSourceRanges: []
# - 10.0.0.0/8
 
# imagePullSecrets allows you to use secrets for pulling images for your pods.
 
imagePullSecrets: []
# - name: "image-pull-secret"
 
# If specified, the pod's nodeSelector，displayName="Map of nodeSelectors to match when scheduling pods on nodes"
 
# Ref: <https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector>
 
nodeSelector:
NodeGroupType: "star-rocks-c9xl"
# <http://kubernetes.io/arch|kubernetes.io/arch>: amd64
 
# <http://kubernetes.io/os|kubernetes.io/os>: linux
 
# affinity for FE proxy pod scheduling.
 
affinity: {}
# nodeAffinity:
 
#   requiredDuringSchedulingIgnoredDuringExecution:
 
#     nodeSelectorTerms:
 
#     - matchFields:
 
#       - key: metadata.name
 
#         operator: In
 
#         values:
 
#         - target-host-name
 
# podAntiAffinity:
 
#   requiredDuringSchedulingIgnoredDuringExecution:
 
#   - labelSelector:
 
#       matchExpressions:
 
#       - key: <http://app.kubernetes.io/component|app.kubernetes.io/component>
 
#         operator: In
 
#         values:
 
#         - fe-proxy
 
#     topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
 
# Node tolerations for FE proxy pod scheduling to nodes with taints
 
# Ref: <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/>
 
tolerations: []
# - key: "key"
 
#   operator: "Equal|Exists"
 
#   value: "value"
 
#   effect: "NoSchedule|PreferNoSchedule|NoExecute(1.6 only)"
 
# Please upgrade the CRD with v1.8.7 released version if you want to use the following configuration.
 
# including: livenessProbeFailureSeconds, readinessProbeFailureSeconds
 
# LivenessProbeFailureSeconds defines the total failure seconds of liveness Probe.
 
# the default value is 15 seconds
 
# You can set it to "0" to disable the probe.
 
livenessProbeFailureSeconds:
# ReadinessProbeFailureSeconds defines the total failure seconds of a readiness Probe.
 
# the default value is 15 seconds
 
# You can set it to "0" to disable the probe.
 
readinessProbeFailureSeconds:
# Note: will create emptyDir volume for FE proxy, PVC is not supported.
 
emptyDirs: []
# e.g., mount an emptyDir volume to /tmp
 
# - name: tmp-data
 
#   mountPath: /tmp
 
# the pod labels for user select or classify pods.
 
podLabels: {}
r
Hi there! I'm Rocky. I took a look at your 
starrocksFeProxySpec
configuration. There are a few items that might cause issues or lead to unexpected behavior in your Kubernetes deployment. Here are the main issues I spotted: 1. Service Ports are Empty (
ports: []
) This is the most critical issue. In your YAML, you have:
Copy code
yaml
service:
  type: ClusterIP
  ports: []
If the 
ports
list is empty, the Kubernetes Service will not expose any ports. The FE Proxy (which is usually an Nginx-based proxy) typically listens on port 8080. You should explicitly define this port so that it can receive traffic. 2. Image Tag is set to 
latest
 Using 
tag: "latest"
combined with 
imagePullPolicy: IfNotPresent
is generally discouraged in production. * Issue: Kubernetes might not pull a newer version of the image if a "latest" image already exists on the node. * Recommendation: Use a specific version tag (e.g., 
v1.8.7
) to ensure consistency across your cluster. 3. Missing Probe Values You have defined the keys for probes but left the values empty:
Copy code
yaml
livenessProbeFailureSeconds:
readinessProbeFailureSeconds:
In YAML, leaving a value blank often assigns it as 
null
. This may cause the Operator to fail validation or use unexpected defaults. If you want the default of 15 seconds, it’s better to either set them to 
15
or remove the lines entirely. 4. Resolver is Empty The FE Proxy uses Nginx, which needs a DNS resolver to find the FE and BE services. * Issue: You've set 
resolver: ""
. * Recommendation: If your cluster uses the standard Kubernetes DNS, set this to 
kube-dns.kube-system.svc.cluster.local
or simply remove the line if the Operator provides a sane default. 5. Single Replica You have 
replicas: 1
. Since the FE Proxy is the entry point for tasks like Stream Load (when using the proxy for redirection), having only one replica creates a single point of failure. Consider increasing this to at least 
2
for high availability. ────────── Recommended Configuration Example Here is a revised version of your spec with these fixes applied:
Copy code
yaml
starrocksFeProxySpec:
  enabled: true
  replicas: 2  # Increased for HA
  imagePullPolicy: IfNotPresent
  image:
    repository: "starrocks/fe-proxy"
    tag: "v1.8.7" # Use a specific version instead of 'latest'
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 1
      memory: 2Gi
  resolver: "kube-dns.kube-system.svc.cluster.local" # Explicitly set the resolver
  service:
    type: ClusterIP
    ports:
      - name: http-port
        port: 8080
        targetPort: 8080 # The default container port for the proxy
  nodeSelector:
    NodeGroupType: "star-rocks-c9xl"
  livenessProbeFailureSeconds: 15
  readinessProbeFailureSeconds: 15
If you are using this proxy specifically to enable Stream Load from outside the cluster, remember that you may need to change the 
service.type
to 
LoadBalancer
or 
NodePort
depending on your network setup! References • deployment/environment_configurations.md • administration/management/FE_configuration.md • administration/stargo.md • quick_start/helm.md
a
@Rocky can feproxyspec proxy the mysql port
r
In the StarRocks Kubernetes Operator and Helm charts, the 
starrocksFeProxySpec
 (often referred to as the FE Proxy) is primarily designed to facilitate Stream Load operations from outside the Kubernetes cluster. Does it proxy the MySQL port? No, the 
starrocksFeProxySpec
does not proxy the MySQL port (default 
9030
). Its main purpose is to solve the "redirection" problem during data loading: 1. When a client sends a Stream Load request to an FE, the FE redirects the client to a BE to upload the data. 2. In a Kubernetes environment, the FE typically returns the internal Pod IP of the BE, which is unreachable from outside the cluster. 3. The FE Proxy (which is an Nginx-based service) handles these HTTP redirections so that external clients can successfully communicate with the BEs. How to proxy or expose the MySQL port To access the MySQL port (
9030
) from outside the cluster, you should configure the service under the 
starRocksFeSpec
 instead:
Copy code
yaml
starrocks:
  starrocksFeSpec:
    service:
      type: LoadBalancer  # Or NodePort, depending on your environment
This will create a Kubernetes Service that exposes the standard StarRocks FE ports, including: • 9030: MySQL query port • 8030: HTTP UI/API port • 9020: RPC port Summary * 
starrocksFeProxySpec
: Used for HTTP-based data loading (Stream Load) and handling BE redirections. * 
starRocksFeSpec.service
: Used for standard MySQL client connections and general cluster access. References • deployment/environment_configurations.md • administration/management/resource_management/Load_balance.md • administration/management/FE_configuration.md • deployment/deploy_manually.md • quick_start/helm.md • deployment/sr_operator.md
a
@Rocky all my fe pods on eks are crashing and trying to come back up, how can I change the helm chart to make one fe node atleast work?
r
If your FE pods are in a 
CrashLoopBackOff
on EKS, it is often due to a quorum failure (the nodes cannot agree on a leader) or metadata corruption. To stabilize the cluster by running exactly one FE node, follow these steps to update your Helm configuration: 1. Scale Replicas to 1 The first step is to reduce the replica count to 1. In a single-node setup, the FE does not need to form a quorum with other nodes to start. Update your 
values.yaml
(or the section in your Helm command):
Copy code
yaml
starrocks:
  starrocksFESpec:
    replicas: 1
2. Force Metadata Recovery (If it still crashes) If the pod continues to crash after scaling to 1, the remaining FE might be stuck waiting for the "old" leader or have a corrupted BDBJE state. You can force this single node to become the leader by adding a recovery flag to the configuration. Depending on your StarRocks version, add one of these to your `values.yaml`: * For v3.1.10+, v3.2.5+, or v3.3+:
Copy code
yaml
    starrocks:
      starrocksFESpec:
        replicas: 1
        config:
          bdbje_reset_election_group: "true"
* For older versions:
Copy code
yaml
    starrocks:
      starrocksFESpec:
        replicas: 1
        config:
          metadata_failure_recovery: "true"
Important: Once the FE node is successfully "Running" and you can connect to it via SQL, you should remove these recovery flags from your Helm chart and run 
helm upgrade
again to return to a normal state. Leaving these flags on can prevent future cluster expansions. 3. Apply the Changes Run the helm upgrade command:
Copy code
bash
helm upgrade <release_name> starrocks/kube-starrocks -f values.yaml
4. How to Debug the Crash If it still won't start, check the logs of the crashing pod to see the specific error:
Copy code
bash
# Get the pod name
kubectl get pods

# Check logs (look for "ERROR" or "FATAL")
kubectl logs <fe-pod-name> -p # -p checks the log of the previous failed instance
Common EKS-specific issues to check: * OOMKilled: Check 
kubectl describe pod <fe-pod-name>
. If you see "OOMKilled", increase the memory requests/limits in 
starrocksFESpec.resources
. * Storage: Ensure your PVCs are correctly bound. If you are not using 
storageSpec
(Persistent Volumes), StarRocks uses 
emptyDir
, meaning all metadata is lost when a pod restarts, which can cause bootstrap loops. * Security Groups/CoreDNS: Ensure the FE pods can resolve each other's hostnames via the headless service. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • quick_start/helm.md
a
@Rocky all 3 fe pod are erroring as soon as they come up
r
Hi, I'm Rocky. If all three FE pods are failing immediately on startup, it's usually due to a networking mismatch, configuration error, or a metadata quorum issue. To help you fix this, I need to see the logs. Please check the output of the following commands: 1. Check the container's standard output:
Copy code
bash
    kubectl logs <fe-pod-name> -n <namespace>
2. Check the internal log files (if the pod stays up long enough to 
exec
in, or by checking the persistent volume): * `fe/log/fe.log`: Main application log. * `fe/log/fe.out`: Standard output/error (often contains JVM startup or fatal crash errors). Common Causes to Look For: 1. 
priority_networks
Mismatch In Kubernetes, FEs often fail because they bind to the wrong network interface. If 
priority_networks
in your 
fe.conf
(or Helm values) doesn't match the Pod's IP range, the FE might bind to a loopback or internal interface that other nodes cannot reach. * What to look for: 
Use IP init local addr, IP: /127.0.0.1
in 
fe.log
. * Fix: Ensure 
priority_networks
is set correctly (e.g., 
10.0.0.0/8
or the specific CIDR of your pod network). 2. JVM Startup Failure If there is an invalid parameter in your 
JAVA_OPTS
, the FE process will die before it even starts logging to 
fe.log
. * What to look for: Check 
fe.out
. You might see 
Unrecognized VM option
or 
Could not create the Java Virtual Machine
. * Fix: Review your memory settings (
-Xmx
, 
-Xms
) and ensure they fit within your Kubernetes resource limits. 3. Metadata Quorum / BDBJE Issues If this is an existing cluster that crashed, the FEs might be unable to elect a leader. * What to look for: 
com.sleepycat.je.rep.InsufficientReplicasException
or 
DatabaseNotFoundException
in 
fe.log
. * Fix: * If this is a new install, try clearing the 
meta
directory on your persistent volumes and restarting. * If this is an existing cluster, check if the IP addresses of the pods changed. If the IPs changed and 
priority_networks
wasn't used, the FEs will look for their old IPs in the metadata and fail to start. 4. Storage Permissions The FE process needs read/write access to the 
meta
directory. * What to look for: 
java.io.IOException: Permission denied
in 
fe.out
or 
fe.log
. Could you share the last 20-30 lines of your 
fe.log
or 
fe.out
? This will let me pinpoint the exact error for you. References • administration/management/FE_configuration.md • administration/Meta_recovery.md • faq/Deploy_faq.md
a
@Rocky what is multi-warehouse mode?
r
In StarRocks, Multi-Warehouse mode is a key feature of the shared-data (cloud-native) architecture. It allows you to create multiple independent compute clusters—referred to as Warehouses or *Compute Groups*—that all access the same underlying data stored in central object storage (like Amazon S3, Google Cloud Storage, or HDFS). This capability decouples computing resources from data, enabling a more flexible and isolated way to manage workloads. Key Characteristics 1. Workload Isolation You can assign specific warehouses to different business units or types of tasks. For example: * Warehouse A for high-frequency data ingestion (ETL). * Warehouse B for interactive BI dashboards. * Warehouse C for heavy ad-hoc data science queries. Since these warehouses are physically separate compute clusters, a massive query in Warehouse C will not impact the performance of the dashboards in Warehouse B. 2. Independent Elasticity Each warehouse can be scaled up or down (adding/removing Compute Nodes) independently based on its specific demand. You can also stop a warehouse when it is not needed to save on costs, without affecting data availability. 3. Data Sharing Even though the compute is isolated, all warehouses see the same "Storage Volume." There is no need to copy or move data between clusters; updates made by one warehouse are immediately visible to others (managed by the shared metadata in the FE). 4. Local Caching Each warehouse maintains its own local hot data cache (on the Compute Nodes' local disks). This ensures that frequently accessed data is served at local-disk speeds, even though the primary data resides in remote object storage. How to Use Multi-Warehouse In a shared-data StarRocks cluster, you can manage and switch between warehouses using SQL commands: * Switching Warehouse: To route your current session's queries to a specific warehouse:
Copy code
sql
    SET warehouse = 'analytics_queries';
    -- or
    USE WAREHOUSE analytics_queries;
* Managing Warehouses: You can create or drop warehouses to manage your compute resources:
Copy code
sql
    CREATE WAREHOUSE reporting_cluster;
    DROP WAREHOUSE old_cluster;
Why Use It? * Prevent Resource Contention: Stop "noisy neighbor" problems where one team's large query slows down everyone else. * Cost Management: Only pay for the compute power each specific task requires. * Simplified Maintenance: Upgrade or restart one warehouse without taking down the entire system. Multi-warehouse mode is primarily available in StarRocks versions 3.0 and later when deployed in the shared-data architecture. References • quick_start/shared-data.md • administration/management/resource_management/resource_group.md • introduction/Architecture.md
a
@Rocky what is the default query_queue_v2_concurrency_level on shared-data mode?
r
In StarRocks, the default value for the FE configuration parameter 
query_queue_v2_concurrency_level
is 4. However, its behavior changes depending on the cluster's running mode: * Shared-nothing mode: The parameter is used directly (default is 4) to calculate total query slots. * Shared-data (multi-warehouse) mode: The effective concurrency level is automatically scaled down to 
max(1, query_queue_v2_concurrency_level / 4)
. Since the default value of the parameter is 4, the effective concurrency level in shared-data/multi-warehouse mode is 1 by default. Key Details: * Total Slots Calculation: This parameter controls the number of logical concurrency "layers." Total slots are derived from 
effective_concurrency_level _ number_of_BEs _ cores_per_BE
. * Purpose: The scaling in multi-warehouse mode ensures that as you add more warehouses (compute clusters) to a shared-data environment, the total concurrent query capacity and resource allocation per slot remain balanced. * Mutability: This is a mutable FE configuration. You can increase it if you need higher concurrent query capacity, but be aware that very large values can reduce the memory allocated per slot and lead to resource fragmentation. You can check the current value in your cluster by running:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE 'query_queue_v2_concurrency_level';
References • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md
a
@Rocky does this parameter tablet_clone_task_timeout_second exist
r
The parameter 
tablet_clone_task_timeout_second
 does not exist in StarRocks. However, there are two very similar parameters in the FE (Frontend) configuration that control the timeout for tablet cloning tasks. Note that they use the suffix 
_sec
instead of `_second`: 1. 
tablet_sched_max_clone_task_timeout_sec
 (Alias: 
max_clone_task_timeout_sec
) * Default: 7200 (2 hours) * Description: The maximum timeout duration for cloning a tablet. 2. 
tablet_sched_min_clone_task_timeout_sec
 (Alias: 
min_clone_task_timeout_sec
) * Default: 180 (3 minutes) * Description: The minimum timeout duration for cloning a tablet. Other Related Timeout Parameters While the cloning parameters use 
_sec
, other tablet-related operations do use the 
_second
suffix: * 
tablet_create_timeout_second
: Timeout for creating a tablet (Default: 10s). * 
tablet_delete_timeout_second
: Timeout for deleting a tablet (Default: 2s). * 
check_consistency_default_timeout_second
: Timeout for replica consistency checks (Default: 600s). If you are trying to adjust the cloning behavior because tasks are timing out, you should look into modifying 
tablet_sched_max_clone_task_timeout_sec
. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md
Open in Slack
PreviousNext
Powered by