http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I have error
# questions-and-troubleshooting
z
@Rocky I have error
r
I'm sorry to hear that. To help you troubleshoot, could you please provide: 1. The exact error message or code you're seeing. 2. The action you were performing (e.g., running a query, loading data, starting a service). 3. Your StarRocks version.
z
@Rocky I think its about starrocks spark connector and can you help with that ?
r
Common issues with the StarRocks Spark connector often involve version mismatches or missing dependencies. To help you troubleshoot, could you share the exact error message? In the meantime, please check if your issue is related to these common causes: 1. Missing MySQL JDBC Driver (Connector 1.1.1+) Starting from version 1.1.1, the Spark connector does not include the MySQL JDBC driver (
mysql-connector-java
) due to licensing. You must manually add it to your Spark classpath (e.g., via 
--jars
). * Symptoms: 
java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver
or 
com.mysql.jdbc.Driver
. 2. Missing 
starrocks.fe.jdbc.url
(Connector 1.1.0+) In version 1.1.0 and later, the connector uses JDBC to fetch table metadata. You must provide the JDBC connection string. * Format: 
jdbc:mysql://<fe_host>:<fe_query_port>
* Common Mistake: Forgetting this parameter or using the HTTP port instead of the query port (default 
9030
). 3. Parameter Name Changes (Connector 1.1.0+) Several parameters were renamed in version 1.1.0. If you upgraded, ensure you are using the new names: * 
starrocks.fenodes
→ 
starrocks.fe.http.url
* 
user
→ 
starrocks.user
* 
password
→ 
starrocks.password
4. Version Mismatch Ensure your connector JAR version matches your Spark and Scala versions. * Example: 
starrocks-spark-connector-3.2_2.12-1.1.0.jar
is for Spark 3.2 and Scala 2.12. If none of these apply, please paste your error log here! References • loading/Spark-connector-starrocks.md • unloading/Spark_connector.md
The error 
ConnectedFailedException
 on port 9060 indicates that your Spark executors cannot reach the StarRocks Backend (BE) nodes at the network level. Based on the address in your error (
kube-starrocks-sn-be-0.kube-starrocks-sn-be-search.starrocks-sn.svc.cluster.local
), it appears you are running StarRocks inside a Kubernetes cluster. Here are the most likely causes and solutions: 1. Network Accessibility (Most Common) Spark executors must have a direct network path to the BEs on the Thrift port (default 
9060
). * If Spark is outside the K8s cluster: It cannot resolve or reach the 
.svc.cluster.local
domain. You must either run Spark inside the same K8s cluster or use a VPN/Peering that allows Spark to reach the K8s pod network. * If Spark is inside the K8s cluster: Ensure it is in a namespace that has access to the 
starrocks-sn
namespace. Check for any NetworkPolicies that might be blocking traffic on port 9060. 2. Verify BE Address in StarRocks The Spark connector asks the FE for a list of BEs, and the FE returns the addresses it has registered. Run this in your MySQL client:
Copy code
sql
SHOW BACKENDS;
Look at the 
IP
(or 
FQDN
) and 
BePort
columns. If the address listed there is not reachable from your Spark environment, the connector will fail. * If you need the BEs to report a different address (e.g., an IP instead of an FQDN), you may need to check your BE configuration for the 
priority_networks
parameter or the StarRocks Operator settings. 3. DNS Resolution If Spark is inside K8s but still failing, try to 
ping
or 
telnet
the host from a Spark worker pod:
Copy code
bash
# From a Spark pod
telnet kube-starrocks-sn-be-0.kube-starrocks-sn-be-search.starrocks-sn.svc.cluster.local 9060
If this fails, your K8s DNS or the headless service (
kube-starrocks-sn-be-search
) is likely misconfigured. 4. Check BE Status Ensure the BE process is actually running and healthy. If the BE is crashing or in a "Down" state, port 9060 will refuse connections. * Check BE logs: Look for errors in 
be.out
or 
<http://be.INFO|be.INFO>
on the BE pod. * Check if the BE is bound to the correct interface (check 
be.conf
for 
be_port
). Summary Checklist 1. Is Spark in the same network as the BE pods? 2. Is port 9060 open? (This is the 
be_port
, not the HTTP port). 3. Does 
SHOW BACKENDS
show the BEs as "Alive"? 4. Is the Spark connector version compatible? (e.g., Spark 3.x needs the 
starrocks-spark-connector-3.x
JAR). If you are using the StarRocks Operator, ensure that the 
starrocks-sn-be-search
service exists and that the BE pods are correctly registered to it. References • unloading/Spark_connector.md • deployment/sr_operator.md • loading/Spark-connector-starrocks.md
z
@Rocky no it is accessible
r
If you've confirmed the BE nodes are reachable, the most likely remaining cause is that the Spark Executors (not just the driver) cannot resolve or reach the BEs directly. In the Spark connector's read path, the connector fetches a list of BE addresses from the FE and then attempts to establish a direct connection to each BE pod. Here are the specific scenarios to check when "accessible" seems true but the error persists: 1. Direct Connectivity from Executors A common pitfall is that the Spark Driver or your local machine can reach the BEs, but the Spark Executor pods cannot. * Verification: Identify a Spark Executor pod and run:
Copy code
bash
  kubectl exec <spark-executor-pod-name> -- telnet kube-starrocks-sn-be-0.kube-starrocks-sn-be-search.starrocks-sn.svc.cluster.local 9060
* Why it fails: If Spark is running in a different namespace or a different subnet without proper DNS/routing to the 
starrocks-sn
namespace, the executors will fail to connect even if the Driver is fine. 2. The "Load Balancer" Trap If you are accessing StarRocks through a Load Balancer or a single Service IP: * The Problem: The Spark connector does not use your Load Balancer for data transfer. It asks the FE for the "real" BE addresses (which in K8s are usually the pod FQDNs you see in your error). * The Solution: The Spark Executors must have a network path to the internal pod IPs/FQDNs of the BEs. If your Spark cluster is outside K8s, it likely cannot resolve 
.svc.cluster.local
. 3. StarRocks Version vs. Connector Version There was a change in how the connector handles connections. Please verify: * Spark Connector version: (e.g., 1.1.1, 1.1.2?) * StarRocks version: (e.g., 3.1, 3.2?) * Requirement: Ensure you are using the JAR specifically built for your Spark version (e.g., 
starrocks-spark-connector-3.2_2.12-1.1.2.jar
for Spark 3.2). 4. BE Port Configuration Verify that the BEs are actually listening on 9060. * Run 
SHOW BACKENDS;
in StarRocks. * Check the 
IP
and 
BePort
columns. If the port listed there is not 9060, or if the IP address listed is one that Spark cannot reach, you need to adjust your 
priority_networks
in 
be.conf
or update the BE registration. 5. Check BE Logs for Rejecting Connections If the network is truly open, the BE might be rejecting the connection. Check the 
<http://be.INFO|be.INFO>
or 
be.out
logs on the pod `kube-starrocks-sn-be-0`: * Look for errors like 
Thrift server client number reaches the limit
. * Check if there are any "Connection reset by peer" or "Handshake failed" messages. To help further, could you tell me: 1. Where is Spark running (same K8s cluster, different K8s cluster, or on-prem/VM)? 2. What is your StarRocks and Spark Connector version? version? References • deployment/sr_operator.md • unloading/Spark_connector.md • administration/management/enable_fqdn.md • loading/Spark-connector-starrocks.md
The error 
out of sequence response: expected X but got Y
is a specific Thrift protocol error. It means the Spark executor sent a request and received a response that doesn't match the expected sequence ID. This typically happens when the connection becomes desynchronized. Since you are in a Kubernetes environment and confirmed the BE is accessible, this is likely caused by one of the following: 1. Service Mesh or L7 Proxy (Most Likely) If your K8s cluster uses a service mesh like Istio or Linkerd, or if you are routing traffic through an L7 Load Balancer (like NGINX Ingress or Envoy): * The Issue: These proxies often attempt to intercept TCP traffic to provide features like retries or multiplexing. Because the StarRocks BE port (
9060
) uses a custom Thrift-over-TCP protocol, the proxy can accidentally interleave packets or misinterpret the stream, causing the "out of sequence" error. * The Solution: Exclude the StarRocks BE ports (
9060
for data, 
8040
for heartbeat) from the service mesh sidecar interception. In Istio, you can use the 
<http://traffic.sidecar.istio.io/excludeOutboundPorts|traffic.sidecar.istio.io/excludeOutboundPorts>
annotation. 2. Network Timeouts and Retries If the network is unstable or the BE is slow to respond: * The Issue: A request might time out on the Spark side, but the BE eventually sends the response. If the Spark executor reuses that same connection for a new request, it might read the "old" response from the first request, leading to a sequence mismatch. * The Solution: * Increase the query timeout in your Spark configuration: 
"starrocks.request.query.timeout.s" = "3600"
(or a higher value). * On the StarRocks BE side, increase the 
scan_context_gc_interval_min
(default is 5 minutes) to prevent the BE from cleaning up the scan state too early. 3. Resource Contention / Thread Safety Though rare in the official connector, this error can occur if multiple threads share the same Thrift client. * The Solution: Ensure you are not manually attempting to share a 
DataFrame
or 
RDD
across threads in a way that triggers concurrent reads on the same underlying connection. Each Spark task should handle its own connection. 4. Version Mismatch Check if there is a significant version gap between your StarRocks cluster and the Spark connector. * StarRocks 3.x is best paired with Connector version 1.1.0 or later. If you are using an older 1.0.x connector with a 3.x cluster, protocol changes might cause instability. Recommended Next Steps: 1. Bypass the Service Mesh: If using Istio/Linkerd, disable it for the Spark-to-BE communication. 2. Check BE Logs: Look at the BE logs (
<http://be.INFO|be.INFO>
) at the exact timestamp of the error. Look for "connection reset" or "timeout" messages. 3. Verify Port: Confirm you are hitting the 
be_port
(9060) and not the 
heartbeat_service_port
(9050) or 
brpc_port
(8060). Are you using Istio or any other sidecar proxy in your K8s pods? References • loading/Spark-connector-starrocks.md • unloading/Flink_connector.md • faq/operation_maintenance_faq.md • unloading/Spark_connector.md References • unloading/Flink_connector.md • unloading/Spark_connector.md • faq/operation_maintenance_faq.md
z
@Rocky E20260807 182121.950606 133139319940672 backend_base.cpp:191] getNext error: context offset [26314 ] ,client offset [ 26263 ] E20260807 182121.951857 133139319940672 backend_base.cpp:191] getNext error: context offset [26314 ] ,client offset [ 26263 ] W20260807 182152.082621 133139319940672 fragment_context.cpp:220] [Driver] Canceled, query_id=019fdc62-19b4-7227-bda6-583ade0c37fc, instance_id=6a47db3e-ef34-60bb-9975-7b303bb65e9d, reason=FragmentContext(id=6a47db3e-ef34-60bb-9975-7b303bb65e9d) cancelled by close_scanner
r
The logs you've provided clarify that the issue isn't just a simple network "connection refused," but a protocol desynchronization between the Spark Connector and the StarRocks Backend (BE). Analysis of the Error 1. Thrift Sequence Mismatch (
expected 196 but got 194
): This is a low-level Apache Thrift error. It means the Spark executor sent a request (ID 196) but received a response for an older request (ID 194). 2. Offset Mismatch (
context offset [26314] , client offset [26263]
): This log from 
backend_base.cpp
confirms the desync. The StarRocks BE is at row/batch 26314, but your Spark executor is asking for data starting from 26263. Most Likely Causes 1. Network Timeouts & Stale Connections (Most Common) If there is a brief network lag or the BE is slow to respond, the Spark connector might "time out" on a specific request but reuse the same TCP connection for the next request. The response for the first request then arrives late and is read as the response for the second request, causing the sequence and offset mismatch. * Solution: Increase the query and connection timeouts in your Spark configuration to prevent the connector from giving up too early:
Copy code
scala
    "starrocks.request.query.timeout.s" = "3600"
    "<http://starrocks.request.connect.timeout.ms|starrocks.request.connect.timeout.ms>" = "30000"
2. Service Mesh / Proxy Interference If you are running in Kubernetes with Istio, Linkerd, or an L7 proxy (like Envoy), these proxies often intercept and "manage" TCP connections. They may attempt to multiplex or retry requests at the proxy level, which breaks the StarRocks Thrift protocol. * Solution: Ensure the StarRocks BE port (
9060
) and FE port (
8030
) are excluded from your service mesh. For Istio, add this annotation to your pods:
Copy code
yaml
    <http://traffic.sidecar.istio.io/excludeOutboundPorts|traffic.sidecar.istio.io/excludeOutboundPorts>: "9060,8030"
3. Resource Pressure (BE or Executor) The log 
cancelled by close_scanner
suggests the BE eventually gave up on this scan context. If the BE is under high memory pressure or the Scan Context GC hits, it will invalidate the session. * Solution: Increase the Scan Context timeout on the StarRocks BE (not Spark). In your 
be.conf
, increase:
Copy code
properties
    scan_context_gc_interval_min = 10  # Increase from default 5
4. Spark Connector Version Bug Older versions of the Spark connector (specifically early 1.1.x versions) had edge cases in how they handled retries and connection pooling. * Check: Ensure you are using the latest stable version of the connector (e.g., 1.1.2 or higher) compatible with your Spark version (3.x). Troubleshooting Steps 1. Verify direct connection: From the specific Spark executor pod that failed, try to run a simple 
telnet
or 
nc
to port 9060 to ensure no intermittent packet loss. 2. Check BE Logs for Crashes: Check 
be.out
on 
kube-starrocks-sn-be-0
to see if the process restarted or hit an OOM (Out of Memory). 3. Reduce Parallelism: As a test, try reducing the number of Spark partitions (
starrocks.request.tablet.size
or 
repartition
) to see if the error disappears when the load on the BEs is lower. To provide more specific guidance, could you share: * The StarRocks version (e.g., 3.1.5) and Spark Connector version you are using. * Whether you have a Service Mesh (Istio/Istio) enabled. References • unloading/Flink_connector.md • sql-reference/sql-statements/table_bucket_part_index/SELECT.md • administration/Meta_recovery.md • knowledge/usage/best-practice.md
r
@Rocky What is the change in AuditLoader v 4.2.1 and v 5.0.0?
r
The primary change between AuditLoader v4.2.1 and v5.0.0 involves a significant update to support newer StarRocks versions and improved compatibility for evolving audit log schemas. Key Changes in AuditLoader v5.0.0 * StarRocks Version Support: AuditLoader v5.0.0 is specifically designed for StarRocks v3.3.11 and later. While older versions like v4.2.1 were standard for previous releases, v5.0.0 is the current recommended version for the StarRocks 3.x and 4.x series. * Java Version Upgrade: AuditLoader v5.0.0 requires Java 11, whereas earlier versions typically ran on Java 8. * Schema Compatibility: The v5.0.0 "new audit plugin" uses a more resilient field-selection mechanism. Because audit log fields (found in 
fe/log/fe.audit.log
) change between major StarRocks versions, v5.0.0 is optimized to select common log fields across versions to prevent loading errors during cluster upgrades. * New Audit Fields: Version 5.0.0 includes support for newer performance and resource-tracking fields introduced in recent StarRocks versions, such as: * 
cpuCostNs
, 
memCostBytes
* 
planCpuCosts
, 
planMemCosts
* 
hitMVs
(Materialized View hits) * 
resourceGroup
* Configuration Improvements: * Password Encryption: Supports the 
secret_key
parameter in 
plugin.conf
to encrypt the database password using the 
AES_ENCRYPT
function. * Data Filtering: Includes a 
filter
parameter that leverages the 
where
condition of Stream Load, allowing you to selectively load only certain audit events (e.g., 
filter=isQuery=1
). Summary Comparison Feature | AuditLoader v4.2.1 | AuditLoader v5.0.0 StarRocks Target | Legacy / Pre-3.3 versions | v3.3.11+ and v4.x Java Version | Java 8 | Java 11 Field Mapping | Static / Version-specific | Dynamic / Common-field selection Encryption | Standard | Supports 
secret_key
encryption For users on StarRocks 3.3.11 or any 4.x version, you should use v5.0.0 to ensure all resource consumption metrics (CPU/Memory) are correctly captured and to avoid plugin failures when new fields are added to the audit. fields are added to the audit logs. References • administration/management/audit_loader.md
r
@Rocky
Copy code
Starrocks > show compute nodes\G;
*************************** 1. row ***************************
        ComputeNodeId: 10023
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.plattestvir05.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-08-10 08:53:05
        LastHeartbeat: 2026-08-10 09:24:35
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 1.88 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 3.5MB/159.8GB, MemUsage: 60.3KB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 315
*************************** 2. row ***************************
        ComputeNodeId: 10024
                   IP: kube-starrocks-cn-1.kube-starrocks-cn-search.plattestvir05.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-06-23 15:05:35
        LastHeartbeat: 2026-06-23 15:05:35
                Alive: false
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg: java.net.UnknownHostException: kube-starrocks-cn-1.kube-starrocks-cn-search.plattestvir05.svc.cluster.local
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 0.00 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: N/A
       HasStoragePath: false
           StatusCode: DISCONNECTED
          StarletPort: 9070
             WorkerId: 2
        WarehouseName: default_warehouse
            TabletNum: 0
2 rows in set (0.00 sec)

ERROR:
No query specified
r
Based on the 
SHOW COMPUTE NODES
output, your StarRocks cluster has a significant connectivity issue that is likely the root cause of the Spark Connector errors you saw earlier. 1. Analysis of the Cluster State * Node 10023 (CN-0): Healthy and 
Alive: true
. * Node 10024 (CN-1): 
Alive: false
 and 
DISCONNECTED
. * Error: 
java.net.UnknownHostException
. This means the StarRocks FE (leader) cannot resolve the DNS name 
kube-starrocks-cn-1.kube-starrocks-cn-search.plattestvir05.svc.cluster.local
. * Time Gap: Its last heartbeat was on 2026-06-23, meaning this node has been unreachable for nearly two months. 2. Connection to the Spark "Out of Sequence" Error The 
expected 196 but got 194
error in Spark occurs when the communication protocol between Spark and the Backend (BE/CN) becomes desynchronized. * If the Spark Connector is still trying to route traffic or metadata through a dead/flapping node (CN-1), or if the FE is returning inconsistent backend lists due to these DNS failures, the connector can end up with stale or corrupted socket streams. * The 
UnknownHostException
confirms that the internal K8s DNS or the Pod lifecycle is broken for that specific node. 3. Recommended Actions Step A: Clean up the dead node If you intentionally scaled down the cluster or if 
kube-starrocks-cn-1
is permanently gone, you must manually remove it from the StarRocks metadata. The FE will continue to try and "talk" to it until you do. Run this on your StarRocks cluster:
Copy code
sql
ALTER SYSTEM DROP COMPUTE NODE "kube-starrocks-cn-1.kube-starrocks-cn-search.plattestvir05.svc.cluster.local:9050";
(Use the 
HeartbeatPort
9050 for the drop command). Step B: Fix K8s DNS/Pods (If the node should be there) If you expect to have two compute nodes: 1. Check Pod Status: 
kubectl get pods -n <namespace>
and look for 
kube-starrocks-cn-1
. 2. Check Service: Ensure the headless service 
kube-starrocks-cn-search
actually has endpoints for both pods. 3. DNS Test: From an FE pod, try to ping the failing FQDN: 
kubectl exec -it <fe-pod> -- ping kube-starrocks-cn-1.kube-starrocks-cn-search.plattestvir05.svc.cluster.local
Step C: Spark Connector Stability Once the cluster state is clean (
Alive: true
for all registered nodes), the "out of sequence" error should subside. If it persists on the remaining healthy node: 1. Ensure no Istio/Service Mesh is intercepting traffic on port 
9060
. 2. Increase the Spark timeout: 
--conf "spark.starrocks.request.query.timeout.s=3600"
. Version Note You are running StarRocks 4.0.1. This is a major version that introduced significant changes to the execution engine and metadata handling. Ensuring a clean 
SHOW COMPUTE NODES
list is critical for the new event-based scheduler in 4.x to function correctly. References • release_notes/release-4.0.md
r
@Rocky Why CN version still showing 4.0.1 whereas I have upgrade both FE and CN. Fe showing correct 4.1.3:
Copy code
Starrocks > show frontends\G;
*************************** 1. row ***************************
               Id: 1
             Name: kube-starrocks-fe-0.kube-starrocks-fe-search.plattestvir05.svc.cluster.local_9010_1782210887512
               IP: kube-starrocks-fe-0.kube-starrocks-fe-search.plattestvir05.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 222987324
             Join: true
            Alive: true
ReplayedJournalId: 1465868
    LastHeartbeat: 2026-08-10 09:24:20
         IsHelper: true
           ErrMsg:
        StartTime: 2026-08-10 08:52:45
          Version: 4.1.3-8a8e186
*************************** 2. row ***************************
               Id: 2
             Name: kube-starrocks-fe-2.kube-starrocks-fe-search.plattestvir05.svc.cluster.local_9010_1782210980030
               IP: kube-starrocks-fe-2.kube-starrocks-fe-search.plattestvir05.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: LEADER
        ClusterId: 222987324
             Join: true
            Alive: true
ReplayedJournalId: 1465870
    LastHeartbeat: 2026-08-10 09:24:20
         IsHelper: true
           ErrMsg:
        StartTime: 2026-08-10 08:52:30
          Version: 4.1.3-8a8e186
*************************** 3. row ***************************
               Id: 3
             Name: kube-starrocks-fe-1.kube-starrocks-fe-search.plattestvir05.svc.cluster.local_9010_1782210980168
               IP: kube-starrocks-fe-1.kube-starrocks-fe-search.plattestvir05.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 222987324
             Join: true
            Alive: true
ReplayedJournalId: 1465868
    LastHeartbeat: 2026-08-10 09:24:20
         IsHelper: true
           ErrMsg:
        StartTime: 2026-08-10 08:52:30
          Version: 4.1.3-8a8e186
3 rows in set (0.03 sec)

ERROR:
r
The fact that 
SHOW COMPUTE NODES
reports 4.0.1-cd9df36 while your FEs are at 4.1.3 indicates that the actual binary running inside your CN pods is still the older version. Even though the node restarted recently (
2026-08-10 08:53:05
), it simply restarted with the old image. In StarRocks, the version displayed in 
SHOW COMPUTE NODES
is sent directly by the CN to the FE during each heartbeat. If it says 4.0.1, it is 4.0.1. 1. Verify the Image Tag in Kubernetes In a Kubernetes deployment (based on your hostnames), this usually happens because the 
StarRocksCluster
CR or your Helm 
values.yaml
was only partially updated. Check the actual image assigned to your CN pods:
Copy code
bash
kubectl get pod kube-starrocks-cn-0 -o jsonpath='{.spec.containers[*].image}'
Compare this with the FE pod image:
Copy code
bash
kubectl get pod kube-starrocks-fe-0 -o jsonpath='{.spec.containers[*].image}'
2. Verify the Binary Inside the Pod To be 100% sure what version the binary thinks it is, run this command to inspect the CN logs or binary directly:
Copy code
bash
kubectl exec -it kube-starrocks-cn-0 -- /opt/starrocks/cn/bin/starrocks_be --version
(Note: The path might vary depending on your Docker image, but it is typically in the 
bin
folder of the installation). 3. Common Reasons for This Discrepancy * Image Tag Not Updated: If you are using the StarRocks Operator, ensure both 
starRocksFeSpec.image
and 
starRocksCnSpec.image
(or 
starRocksBeSpec.image
) are updated to 
4.1.3
. * Rollout Failure: The CN pods might have failed to pull the new 4.1.3 image (e.g., due to an 
ImagePullBackOff
), causing them to stick with the old version or fail to update. * Helm Values: If using Helm, you might have overridden the CN image version specifically in your 
values.yaml
while the FE used a global default. 4. Cleanup Recommendation While you are fixing the versions, you should also remove the dead node 
10024
which is still in your metadata from June. Leaving dead nodes in the list can cause the FE to waste resources trying to send heartbeats and can confuse the Spark Connector. Run this to drop the dead node:
Copy code
sql
ALTER SYSTEM DROP COMPUTE NODE "kube-starrocks-cn-1.kube-starrocks-cn-search.plattestvir05.svc.cluster.local:9050";
Recommendation: Update your K8s configuration to ensure the CN pods are using the 
4.1.3
image tag and perform a 
kubectl rollout restart statefulset <cn-statefulset-name>
to force the update if the pods didn't update automatically. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • deployment/upgrade.md • quick_start/helm.md • administration/management/BE_configuration.md References • faq/Others.md • faq/operation_maintenance_faq.md • quick_start/helm.md • deployment/deploy_manually.md
z
@Rocky in you official helm chart UpdateStrategy of FE is rollingupdate and is it ok to change to OnDelete? because as you said order matters while upgrading to a newer version and it must be first followers than leader and it cant be made via rollingupdate it does not guarantees that leader will be updated last
r
The official StarRocks documentation and best practices do indeed emphasize that for a successful upgrade, Follower FEs must be upgraded before the Leader FE. If the Leader is upgraded first, it can lead to metadata incompatibility and cluster instability. Regarding your question about changing the 
UpdateStrategy
to `OnDelete`: 1. Is it okay to change to 
OnDelete
? Yes, it is common and often recommended for production FE clusters. By changing the 
UpdateStrategy
to 
OnDelete
in your Helm 
values.yaml
(or within the 
StarRocksCluster
CR if using the Operator), you prevent Kubernetes from automatically restarting FE pods in an arbitrary or ordinal order that might not respect the Leader's role. * RollingUpdate (Default): Kubernetes updates pods in reverse ordinal order (e.g., 
fe-2
, then 
fe-1
, then 
fe-0
). It has no awareness of which pod is the StarRocks Leader. If 
fe-2
is the Leader, it will be updated first, which violates the upgrade rule. * OnDelete: Kubernetes will only update a pod to the new version (e.g., new image tag) when you manually delete that specific pod. This gives you absolute control over the sequence. 2. How to upgrade with 
OnDelete
 If you switch to 
OnDelete
, follow this workflow: 1. Update the Image: Change the image version in your Helm 
values.yaml
and run 
helm upgrade
. Kubernetes will update the StatefulSet specification but will not restart the pods. 2. Identify the Leader: Run 
SHOW FRONTENDS;
in your MySQL client to see which node has the 
Role: LEADER
. 3. Delete Followers First: Manually delete the pods that are 
FOLLOWERS
one by one.
Copy code
bash
    kubectl delete pod <follower-pod-name>
Wait for the pod to come back up and join the cluster (
Alive: true
) before moving to the next. 4. Delete the Leader Last: Once all Followers are running the new version, delete the Leader pod. A new Leader (already on the new version) will be elected from the healthy Followers, and the old Leader pod will restart with the new version. 3. Does the StarRocks Operator handle this? While the StarRocks Operator (used by the 
kube-starrocks
chart) is "aware" of the cluster, it generally relies on the underlying Kubernetes 
UpdateStrategy
. Some advanced versions of the operator may attempt to manage this, but using 
OnDelete
remains the most reliable "safety first" method to ensure the metadata upgrade path is followed strictly. Summary Recommendation * For Development/Test: 
RollingUpdate
is usually fine as a quick restart often resolves election issues, though not ideal. * For Production: Use 
OnDelete
for the FE component. It ensures you can verify metadata synchronization (via 
fe.log
and 
SHOW FRONTENDS
) at every step before touching the Leader. References • deployment/upgrade.md • deployment/helm.md • faq/operation_maintenance_faq.md References • deployment/upgrade.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md
r
@Rocky Can i run FE pods on amd ec2 instance and CN on arm64 ec2 instance ?
r
Technically, yes, you can run FE pods on AMD64 (x86) and CN pods on ARM64 (Graviton) in the same cluster, but it is not recommended for production due to hardware heterogeneity and supportability. 1. Technical Feasibility * Protocol Independence: The communication between the Frontend (FE) and Compute Node (CN) uses bRPC with Protocol Buffers, which is architecture-agnostic. * Java-based FE: The FE is a Java application. As long as you use a multi-arch Docker image or the correct arch-specific image, the FE logic remains the same across AMD and ARM. * Binary Availability: StarRocks provides official binaries and Docker images for both 
amd64
(x86_64) and 
arm64
. 2. Official Recommendations & Risks While it works technically, you should be aware of the following official stances and risks: * Heterogeneity Warning: StarRocks documentation explicitly states that "Heterogeneous deployments are not supported" for Compute Nodes. This means all CN nodes within a cluster should have the same hardware specifications (including architecture) to avoid performance bottlenecks and unpredictable behavior in the MPP (Massively Parallel Processing) execution engine. * Performance Imbalance: x86 architectures utilize AVX2/AVX-512 instruction sets for vectorization, while ARM64 uses NEON. The execution speeds for the same query fragments may differ between architectures, which can lead to "long-tail" latency in distributed queries where the FE has to wait for the slowest node to finish. * Troubleshooting Complexity: If you encounter bugs related to the execution engine or memory management, the StarRocks community and support will likely ask you to reproduce the issue on a homogenous cluster first. 3. Implementation in Kubernetes If you decide to proceed (e.g., for cost optimization in a dev environment), you must ensure your Kubernetes configuration routes the pods correctly: 1. Use Multi-Arch Images: Use the standard StarRocks images (e.g., 
starrocks/fe-ubuntu:4.1.3
). Kubernetes will automatically pull the correct architecture version for the node it lands on. 2. Node Affinity: Use 
nodeSelector
or 
affinity
in your 
StarRocksCluster
CR (if using the Operator) to ensure the FE and CN land on the intended instance types:
Copy code
yaml
    # Example for CN Spec
    starRocksCnSpec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            • matchExpressions:
              • key: <http://kubernetes.io/arch|kubernetes.io/arch>
                operator: In
                values:
                • arm64
Summary Recommendation * Dev/Test: It is fine to mix them to save costs (e.g., using Graviton for compute). * Production: Stick to a homogenous architecture. Use either all AMD64 or all ARM64 for both FE and CN/BE to ensure stability and predictable performance. References • deployment/prepare_deployment_files.md • faq/Deploy_faq.md • deployment/plan_cluster.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md References • developers/mac-compile-run-test.md • deployment/deployment_prerequisites.md • developers/build-starrocks/Build_in_docker.md
r
@Rocky I want to give access to create table and insert data into it only in a specific database. What permission need to give to thta user role?
Copy code
Starrocks > GRANT INSERT ON DATABASE abc_migration TO ROLE 'sr_redshift_role';
ERROR 1064 (HY000): Getting analyzing error. Detail message: Cannot grant or revoke INSERT on 'DATABASE' type object.
Starrocks >
r
The error occurred because 
INSERT
 is a table-level privilege, while your command attempted to apply it to a 
DATABASE
 object. In StarRocks, privileges are categorized by the type of object they act upon. To allow a role to both create new tables and insert data into tables within a specific database, you need to run two separate grants: 1. Grant Permission to Create Tables The 
CREATE TABLE
privilege belongs to the 
DATABASE
object.
Copy code
sql
GRANT CREATE TABLE ON DATABASE abc_migration TO ROLE 'sr_redshift_role';
2. Grant Permission to Insert Data The 
INSERT
privilege belongs to the 
TABLE
object. To grant this for all tables within that database, use the 
ALL TABLES IN DATABASE
syntax:
Copy code
sql
GRANT INSERT ON ALL TABLES IN DATABASE abc_migration TO ROLE 'sr_redshift_role';
3. Grant Catalog Usage (Recommended) In StarRocks 3.x and later, you usually need the 
USAGE
privilege on the catalog (the default is 
default_catalog
) to access the databases within it:
Copy code
sql
GRANT USAGE ON CATALOG default_catalog TO ROLE 'sr_redshift_role';
Summary of Correct Workflow
Copy code
sql
-- 1. Allow using the catalog
GRANT USAGE ON CATALOG default_catalog TO ROLE 'sr_redshift_role';

-- 2. Allow creating new tables in the database
GRANT CREATE TABLE ON DATABASE abc_migration TO ROLE 'sr_redshift_role';

-- 3. Allow inserting into all existing and future tables in that database
GRANT INSERT ON ALL TABLES IN DATABASE abc_migration TO ROLE 'sr_redshift_role';
Note: If a user creates a table themselves, they are automatically granted all privileges (including 
INSERT
) on that specific table. However, granting 
INSERT ON ALL TABLES
is necessary if they need to insert data into tables created by other users or if you want to ensure they have continuous access. References • sql-reference/sql-statements/account-management/GRANT.md • administration/user_privs/authorization/privilege_item.md • administration/user_privs/authorization/User_privilege.md References • administration/user_privs/authorization/User_privilege.md • sql-reference/sql-statements/account-management/GRANT.md • administration/user_privs/authorization/privilege_item.md
r
@Rocky I also need seelct:
Copy code
Starrocks > select * from test_table;
ERROR 5203 (42000): Access denied; you need (at least one of) the SELECT privilege(s) on TABLE test_table for this operation. Please ask the admin to grant permission(s) or try activating existing roles using <set [default] role>. Current role(s): [sr_redshift_role]. Inactivated role(s): NONE.
Starrocks >
This is what i have:
Copy code
Starrocks > SHOW GRANTS FOR ROLE sr_redshift_role;
+------------------+-----------------+-----------------------------------------------------------------------------------+
| UserIdentity     | Catalog         | Grants                                                                            |
+------------------+-----------------+-----------------------------------------------------------------------------------+
| sr_redshift_role | default_catalog | GRANT INSERT ON TABLE quickstart.glue_load_test TO ROLE 'sr_redshift_role'        |
| sr_redshift_role | default_catalog | GRANT INSERT ON ALL TABLES IN DATABASE tempo_migration TO ROLE 'sr_redshift_role' |
| sr_redshift_role | default_catalog | GRANT CREATE TABLE ON DATABASE abc_migration TO ROLE 'sr_redshift_role'         |
| sr_redshift_role | default_catalog | GRANT FILE ON SYSTEM TO ROLE 'sr_redshift_role'                                   |
| sr_redshift_role | default_catalog | GRANT USAGE ON CATALOG default_catalog TO ROLE 'sr_redshift_role'                 |
+------------------+-----------------+-----------------------------------------------------------------------------------+
5 rows in set (0.00 sec)
r
To resolve the access denied error for 
SELECT
, you need to grant the 
SELECT
 privilege at the table level. Similar to the 
INSERT
privilege, you can grant it for all tables within your specific database. Run the following command:
Copy code
sql
GRANT SELECT ON ALL TABLES IN DATABASE abc_migration TO ROLE 'sr_redshift_role';
Why this is happening: * Privilege Granularity: In StarRocks, 
CREATE TABLE
(Database level) allows you to create the object, but it does not automatically grant you 
SELECT
or 
INSERT
on existing tables in that database unless you are the owner or have been granted those specific table-level privileges. * Current State: Your 
SHOW GRANTS
output shows you have 
CREATE TABLE
on 
abc_migration
, but no 
SELECT
or 
INSERT
privileges for that specific database yet (you only have 
INSERT
on 
tempo_migration
). Recommended Full Permission Set for `abc_migration`: To ensure the user can fully manage data within the 
abc_migration
database (Create, Insert, and Query), your role should have these three core grants:
Copy code
sql
-- 1. Create the tables
GRANT CREATE TABLE ON DATABASE abc_migration TO ROLE 'sr_redshift_role';

-- 2. Insert data into the tables
GRANT INSERT ON ALL TABLES IN DATABASE abc_migration TO ROLE 'sr_redshift_role';

-- 3. Query the data (Fixes your current error)
GRANT SELECT ON ALL TABLES IN DATABASE abc_migration TO ROLE 'sr_redshift_role';
Note on "Active" Roles Your error message shows 
Current role(s): [sr_redshift_role]
, which means the role is already active. Once you run the 
GRANT SELECT
command above, the change should take effect immediately for the user. References • sql-reference/sql-statements/account-management/GRANT.md • administration/user_privs/authorization/privilege_item.md • administration/user_privs/authorization/User_privilege.md
r
@Rocky Since I am using AWS Glue job to trigger the Load Command, what if job filed. How will I make sure when I re-trigger , the job starts from the last fialed step?
r
To ensure your AWS Glue job resumes correctly from the last failed step when interacting with StarRocks, you need to handle state at two levels: StarRocks' Idempotency (Labels) and AWS Glue's State Management (Bookmarks/Workflows). 1. StarRocks Level: Using "Labels" for Idempotence StarRocks uses a Label system to ensure "Exactly-Once" semantics. Every load command (
BROKER LOAD
, 
STREAM LOAD
, or 
INSERT INTO
) should be assigned a unique label. * How it works: If a load job fails, StarRocks marks that label as 
CANCELLED
. If you re-trigger the job with the same label, StarRocks knows it is a retry of the previous attempt. If a label is already 
FINISHED
, StarRocks will reject any new load with the same name, preventing duplicate data. * Implementation in Glue: * Generate a label based on the Glue Job Run ID or a timestamped batch ID (e.g., 
abc_migration_20231027_batch1
). * If your Glue job script crashes and retries, logic should ensure it uses the same label for that specific batch. * Note: StarRocks loads are atomic. A single 
LOAD
command either succeeds entirely or fails entirely. It won't "partially" load files and stop in the middle of one; it will roll back the transaction. 2. Glue Level: Using Job Bookmarks If your Glue job processes many files from S3 and you want to ensure that a re-run only picks up the files that weren't successfully loaded yet, use Glue Job Bookmarks. * How it works: Glue stores state information about the source (e.g., which S3 files have been processed). When the job is re-run, it skips the files it already "bookmarked." * Implementation: * Enable Job Bookmarks in the Glue Job configuration. * In your script, use 
job.init()
and 
job.commit()
. * Crucial Step: Ensure the StarRocks 
LOAD
command is successful before calling 
job.commit()
. If the StarRocks load fails, the script should exit with an error so that Glue does not commit the bookmark for those files. 3. Orchestration: AWS Glue Workflows or Step Functions If your Glue "job" actually consists of multiple distinct steps (e.g., Step 1: Data Cleaning, Step 2: Load Table A, Step 3: Load Table B), and you want to skip Step 1 and 2 if Step 3 fails: * Glue Workflows: You can use the AWS Glue Console to "Resume" a workflow. You select the failed node (Step 3) and click Resume. Glue will only run that node and its downstream dependencies. * AWS Step Functions: This is a more robust way to orchestrate. You can define a "Retry" strategy for each step or a "Catch" block that saves the current state to a DynamoDB table, allowing the next execution to read where it left off. Summary Checklist for your Glue Script: 1. Enable Job Bookmarks to avoid re-processing S3 files. 2. Define a Unique Label for the StarRocks 
LOAD
command (e.g., 
label = f"glue_load_{job_run_id}"
). 3. Use a Try-Except Block:
Copy code
python
    try:
        # Trigger StarRocks Load with specific label
        run_starrocks_load(label=my_label)
        # Only commit if StarRocks confirms success
        job.commit()
    except Exception as e:
        print(f"Load failed: {e}")
        raise e # Ensure Glue marks the run as FAILED
References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/loading_introduction/loading_concepts.md • faq/loading/Loading_faq.md References • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • faq/loading/Loading_faq.md • loading/StreamLoad.md
r
@Rocky If I want to conect to FE Kubernetes service from AWS Glue , what its the recommended way. And how the flow will work when Glue job triggers the Load Label command?
r
To connect AWS Glue to a StarRocks cluster running on Kubernetes (EKS), the recommended approach involves setting up secure private networking and using an Internal Load Balancer. 1. Recommended Connection Way Since AWS Glue runs in a managed VPC, you should expose the StarRocks FE (Frontend) via an Internal Network Load Balancer (NLB). * Service Configuration: In your Kubernetes manifest (or Helm values), set the FE service type to 
LoadBalancer
and use AWS-specific annotations to make it internal.
Copy code
yaml
    # Example snippet for starrocks-fe-service
    metadata:
      annotations:
        <http://service.beta.kubernetes.io/aws-load-balancer-internal|service.beta.kubernetes.io/aws-load-balancer-internal>: "true"
        <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: "nlb"
    spec:
      type: LoadBalancer
* Networking: * If Glue and EKS are in the same VPC, Glue can access the NLB directly. * If they are in different VPCs, you must set up VPC Peering or a Transit Gateway. * Security Groups: Ensure the FE's security group allows inbound traffic from the Glue Job's security group on ports 9030 (MySQL protocol for commands) and 8030 (HTTP protocol for Stream Loads). ────────── 2. The Flow of the "Load Label" Command When your Glue job triggers a load (e.g., a 
BROKER LOAD
), the process follows an asynchronous coordination flow: 1. Job Submission (Glue → FE): The Glue job connects to the FE via the NLB using a MySQL client or JDBC. It sends the 
LOAD
command, including a unique 
LABEL
(e.g., 
LABEL my_glue_sync_001
). 2. Plan Generation (FE): The FE checks the 
LABEL
. If the label already exists and was successful, it rejects the job (preventing duplicates). If it's a new or previously failed label, the FE analyzes the S3 path provided in the command and generates a query plan. 3. Task Distribution (FE → BE): The FE distributes the load tasks to the Backend (BE) or Compute Nodes (CN) in your Kubernetes cluster. 4. Parallel Data Pull (BE → S3): The BE nodes pull data directly from S3 in parallel. They do not route the data through the FE. This is why BE nodes must have an IAM role/policy that allows 
s3:GetObject
and 
s3:ListBucket
. 5. Commit and Acknowledgment (BE → FE → Glue): Once all BEs finish reading their portion of the data, they report back to the FE. The FE commits the transaction, marks the 
LABEL
as 
FINISHED
, and makes the data visible for queries. The Glue job receives a "Success" message (for synchronous loads) or can poll 
SHOW LOAD
to check the status. Summary of Benefits * Performance: Data flows directly from S3 to BEs, bypassing the Glue worker's memory limits and the FE's bottleneck. * Idempotency: The 
LABEL
ensures that if your Glue job retries due to a network glitch, StarRocks won't load the same data twice. References • deployment/sr_operator.md • loading/BrokerLoad.md • loading/loading_introduction/loading_concepts.md • integrations/authenticate_to_aws_resources.md References • loading/SparkLoad.md • loading/StreamLoad.md • loading/BrokerLoad.md • loading/Loading_data_template.md • loading/loading_introduction/loading_concepts.md • loading/RoutineLoad.md • loading/hdfs_load.md
2 Views
Open in Slack
PreviousNext
Powered by