http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am running starrocks cluster usin...
# questions-and-troubleshooting
s
@Rocky I am running starrocks cluster using starrocks kubernetes operator and the process is starting with root inside the pod and my cluster is using pvc for the meta storage how I can ensure process need to start with non root user
r
To run StarRocks as a non-root user when using the StarRocks Kubernetes Operator, you can use the built-in 
runAsNonRoot
field and the 
podSecurityContext
in your 
StarRocksCluster
Custom Resource (CR). When using PVCs for meta storage (FE) or data storage (BE), simply running as a non-root user may cause "Permission Denied" errors because Kubernetes often mounts volumes as 
root
. You must also set the 
fsGroup
to ensure the mounted volumes are writable by the non-root process. Configuration Steps Add the following fields to your 
StarRocksCluster
YAML under the component specifications (
starRocksFeSpec
, 
starRocksBeSpec
, etc.):
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-cluster
spec:
  starRocksFeSpec:
    # 1. Enforce non-root execution
    # This automatically sets runAsUser: 1000 and runAsGroup: 1000
    runAsNonRoot: true
    
    # 2. Ensure PVC permissions for meta storage
    podSecurityContext:
      fsGroup: 1000
      
    storageSpec:
      name: fe-meta
      storageSize: 10Gi
      # ... other storage settings
    # ... other FE settings

  starRocksBeSpec:
    runAsNonRoot: true
    podSecurityContext:
      fsGroup: 1000
    # ... other BE settings
Key Parameters: * 
runAsNonRoot: true
: When enabled, the operator configures the container to run as a normal user (typically UID/GID 
1000
). If your container image defaults to root, the operator will override it with these IDs. * 
podSecurityContext.fsGroup: 1000
: This is a standard Kubernetes setting. It tells the Kubelet to change the ownership of the mounted volumes (PVCs) to GID 
1000
, allowing your non-root process to write metadata and data. * 
fsGroupChangePolicy
: (Optional) In newer Kubernetes versions, you can add 
fsGroupChangePolicy: "OnRootMismatch"
inside 
podSecurityContext
to speed up the mounting process for large volumes. Important Notes: * Upgrade Compatibility: Ensure you are using a recent version of the StarRocks Operator (v1.8.0 or later recommended) as support for these security fields has been enhanced in newer releases. * Existing PVCs: If you are switching an existing cluster from root to non-root, Kubernetes will attempt to change permissions on the existing data in the PVC. Depending on the volume size and your storage driver, this might take time during the first restart. References • administration/management/FE_configuration.md • faq/Deploy_faq.md
s
@Rocky same will applicable for cn pod as well and my cn pod doesn't have any pvc
r
Yes, the same configuration applies to the CN (Compute Node) pods. Even if your CN pods do not use PVCs (as they are typically stateless), you should still configure these fields to ensure the process starts with a non-root user and has the necessary permissions for internal operations. Configuration for CN Pods You can add the 
runAsNonRoot
and 
podSecurityContext
fields directly under the 
starRocksCnSpec
section in your YAML:
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-cluster
spec:
  starRocksCnSpec:
    # 1. Start the CN process as a non-root user (UID 1000)
    runAsNonRoot: true
    
    # 2. Recommended for log and ephemeral storage permissions
    podSecurityContext:
      fsGroup: 1000
    
    # ... other CN settings (image, requests, limits, etc.)
Why this is needed for CN (even without PVCs): 1. Process Identity: Setting 
runAsNonRoot: true
tells the Kubernetes Operator to enforce the container's security context. It will set 
runAsUser: 1000
and 
runAsGroup: 1000
automatically, ensuring the StarRocks binary does not run as root. 2. Log Writing: CN pods still write logs to 
/opt/starrocks/cn/log
. While these might be on the container's local writable layer (instead of a PVC), the process still needs appropriate permissions. 3. Spill-to-Disk: If you enable intermediate result "spilling" for large queries, the CN node may write temporary data to local paths. Using 
fsGroup: 1000
ensures that any volumes mounted (even 
emptyDir
volumes for scratch space) are accessible by the non-root user. 4. Consistency: Applying the same security policy across FE, BE, and CN components ensures your entire cluster adheres to your security requirements. Verification After applying the change, you can verify the user by running:
Copy code
bash
kubectl exec -it <cn-pod-name> -- id
The output should show 
uid=1000(starrocks) gid=1000(starrocks)
instead of root. References • deployment/sr_operator.md • faq/Deploy_faq.md References • deployment/sr_operator.md • faq/Deploy_faq.md • deployment/post_deployment_setup.md
s
@Rocky I want to send the logs to open search using fluent bit how I can achieve this?
r
To send StarRocks logs to OpenSearch using Fluent Bit, you need to configure Fluent Bit to tail the log files, handle multi-line patterns (like Java stack traces in FE), and forward them to your OpenSearch endpoint. Since you are using the StarRocks Operator, you have two main approaches: 1. Collect from Container Stdout (Recommended): If you configure StarRocks to log to the console, Fluent Bit can pick them up from the standard Kubernetes log path (
/var/log/containers/*.log
). 2. Collect from Persistent Volumes: If your logs are only in the PVC, you must mount that same PVC to Fluent Bit or use a sidecar. 1. Fluent Bit Configuration You will need a configuration with three parts: 
INPUT
(to read logs), 
FILTER
(to handle multi-line), and 
OUTPUT
(to send to OpenSearch). A. FE Multiline Parser (Java Format) FE logs (
fe.log
) use a standard Java log4j format. Add this to your Fluent Bit `parsers.conf`:
Copy code
text
[PARSER]
    Name        sr_fe_multiline
    Format      regex
    Regex       ^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S,%L
B. BE/CN Multiline Parser (Glog Format) BE and CN logs (
<http://be.INFO|be.INFO>
) use the glog format. Add this to your `parsers.conf`:
Copy code
text
[PARSER]
    Name        sr_be_multiline
    Format      regex
    Regex       ^([IWEF]\d{4} \d{2}:\d{2}:\d{2}.\d{6})
    Time_Key    time
    Time_Format %m%d %H:%M:%S.%L
C. Main Fluent Bit Config In your 
fluent-bit.conf
, use the 
multiline
filter to merge stack traces:
Copy code
yaml
[INPUT]
    Name              tail
    Tag               starrocks.fe
    Path              /var/log/containers/*starrocks-fe*.log # Or your PVC path
    Multiline.Parser  sr_fe_multiline

[INPUT]
    Name              tail
    Tag               <http://starrocks.be|starrocks.be>
    Path              /var/log/containers/*starrocks-be*.log # Or your PVC path
    Multiline.Parser  sr_be_multiline

[OUTPUT]
    Name            opensearch
    Match           starrocks.*
    Host            opensearch-cluster-service
    Port            9200
    HTTP_User       admin
    HTTP_Passwd     your_password
    Index           starrocks-logs
    tls             On
    tls.verify      Off
    Suppress_Type_Name On
2. Ensure Logs are Accessible If you want Fluent Bit to collect logs directly from the file system (rather than K8s stdout), you must ensure your 
StarRocksCluster
CR has log persistence enabled so the volumes can be shared:
Copy code
yaml
spec:
  starRocksFeSpec:
    storageVolumes:
      - name: fe-log
        mountPath: /opt/starrocks/fe/log
        storageSize: 10Gi
3. StarRocks Logging to Console For the easiest integration with Fluent Bit, you can tell StarRocks to log to the console. You can do this by adding environment variables to your 
StarRocksCluster
spec: * FE: The FE process logs to 
fe.out
by default in the container. The operator usually pipes this to stdout. * BE/CN: To ensure all glog output goes to the console, you can try setting the environment variable 
GLOG_logtostderr=1
in the 
beEnvVars
or 
cnEnvVars
section, though the standard behavior of the Operator-managed pods is already to capture the 
.out
files. Summary of Steps: 1. Deploy OpenSearch and ensure it's reachable from the StarRocks namespace. 2. Deploy Fluent Bit as a DaemonSet. 3. Apply Multiline Parsers to Fluent Bit so that Java/C++ stack traces appear as a single document in OpenSearch rather than multiple lines. 4. Point the Output to your OpenSearch index. References • administration/management/logs.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md -configuration)
s
@Rocky I am using the cluster migration tool to migrate the data from ec2 starrocks to eks I can see the tables and databases but while query the table it shows zero results
r
If you are using the StarRocks Cross-cluster Data Migration Tool (the tool provided for StarRocks-to-StarRocks migration) and you can see the table structures but the queries return 0 rows, it usually means the metadata synchronization succeeded, but the data replication jobs are either pending, failed, or the target cluster is not correctly configured to receive the data. Follow these troubleshooting steps to identify why the data is missing: 1. Check "Legacy Compatibility" Configuration The most common reason for data not appearing in the target cluster is that the target (EKS) cluster has not enabled legacy compatibility. This is required for the source cluster to replicate data versions to the target. Run this on your target (EKS) FE:
Copy code
sql
-- Check if enabled
ADMIN SHOW FRONTEND CONFIG LIKE 'enable_legacy_compatibility_for_replication';

-- If it returns false, enable it dynamically
ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication" = "true");
_Note: You should also add 
enable_legacy_compatibility_for_replication = true
to your 
fe.conf
in EKS to ensure it persists after a restart._ 2. Verify Migration Transaction Status The migration tool uses transactions to sync data. If these transactions are stuck in a 
PENDING
or 
PREPARE
state, the data won't be visible in the tables yet. Run this on your target (EKS) cluster:
Copy code
sql
SHOW PROC "/transactions/<your_db_name>/running";
Look for the 
State
column. If there are many running transactions but none are committing, check the 
log/sync.INFO.log
file in the migration tool directory for specific error messages (e.g., connection timeouts or permission errors). 3. Compare Partition Data Versions You can see if any data has actually arrived at the partition level.
Copy code
sql
SHOW PARTITIONS FROM <table_name>;
Check the 
VisibleVersion
column. In the target cluster, if the version is 
1
, it means no data has been synchronized yet. Compare this with the version on your source (EC2) cluster. If the source is at version 
100
and the target is at 
1
, the data sync hasn't started or is failing. 4. Check Connectivity and Ports For data to move, the Source BEs must be able to push data to the Target BEs, and the Migration Tool must be able to talk to both FEs. Ensure these ports are open between your EC2 and EKS environments: • FE: 9030 (Query), 8030 (HTTP), 9020 (RPC) • BE: 8040 (HTTP), 9060 (RPC) 5. Check Migration Tool Logs If the tool is still running, check its logs on the machine where you executed it: • Path: 
log/sync.INFO.log
• Look for 
Sync job progress
. If it shows 
0%
or shows a high number of 
failed
jobs, the logs will contain the specific reason (e.g., "Failed to send replication task"). Summary of common causes: • Only DDL was run: If you used the SMT (StarRocks Migration Tool) and only ran the 
starrocks-create.sql
script, you have only created the schema. You must also run the Flink job (if using SMT) or keep the Cross-cluster tool running to perform the data sync. • Version Mismatch: Ensure the target cluster version is v3.1.8, v3.2.3, or later, as required by the migration tool. • Firewall/Security Groups: EKS worker nodes must allow inbound traffic from EC2 BE IPs on the ports mentioned above. References • administration/data_migration_tool.md • integrations/loading_tools/SMT.md • administration/management/FE_configuration.md References • integrations/loading_tools/SMT.md
2 Views
Open in Slack
PreviousNext
Powered by