http://starrocks.io logo
Join Slack
Powered by
Hi Team, I am new to Starrocks and trying to deplo...
# questions-and-troubleshooting
r
Hi Team, I am new to Starrocks and trying to deploy it in my AWS EKS. I came across this doc https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/examples/starrocks/deploy_a_starrocks_cluster_with_all_features.yaml and got confused. Can we deploy all 3 nodes in same cluster FE+BE+CN? What will it be then Shared-nothing or shared-everything? Can someone guide me on this.
a
In Shared-Nothing mode, you should deploy: • FE Nodes • BE Nodes In Shared-Data mode, you should deploy: • FE Nodes • CN Nodes • BE Nodes should be disabled / not used in this mode. • If you plan to use the Shared-Data architecture, please refer to this guide document (this guide doucment)
r
@ahn heesuk Yes, that was my understanding as well, but this example YAML is confusing me. Does this mean the example is incorrect?
a
I think it is just example. could you check this for shared data mode - shared_data_mode
r
Yes this works fine as well. My doubt is is it a right way to use all 3 to get benefit from all 3 types?
a
as far as I know, we can choice only 
shared nothing [ FE + BE ]
or 
shared data [ FE + CN ]
running all three types of nodes together (a hybrid setup) is not a valid option. 👀
✅ 1
r
I deployed FE and BE and created some tables and inserted few records.Now how I can test whether the whole setup is working as expected like BE is performing its required task or not?
a
In my case, I connected to the 
fe-service
on port 
9030
using a MySQL client and checked whether queries were working properly.
And if you access port 
8030
over HTTP, you can view a simple admin dashboard to check various status information.
r
@ahn heesuk I am trying to deploy FE+CN nodes using helm chart. Both nodes are up and healthy but when I am trying to create a table, I am seeing this error:
Copy code
mysql> CREATE TABLE IF NOT EXISTS crashdata ( COLLISION_ID INT, CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ) DUPLICATE KEY(COLLISION_ID) DISTRIBUTED BY HASH(COLLISION_ID) BUCKETS 3 PROPERTIES( "storage_volume" = "S3VOL" ); ERROR 1064 (HY000): Getting analyzing error. Detail message: Cluster has no available capacity.
This is my values.yaml:
Copy code
metrics:
  serviceMonitor:
    enabled: true
starrocksCluster:
  # the namespace of starrockscluster cluster, if not set, the release namespace will be used.
  namespace: "dockprivileged"
  # specify the BE/CN deployment or not.
  enabledBe: false
  enabledCn: true
initPassword:
  enabled: true
  passwordSecret: starrocks-root-pass
starrocksFESpec:
  service:
    type: ClusterIP
    ports:
      - name: query
        port: 9030
      - name: http
        port: 8030
  # number of replicas to deploy for a FE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/fe-ubuntu
    tag: ""
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  serviceAccountName: kubvir-starrocks-s3-sa
  config: |
    enable_load_volume_from_conf=true
    cloud_storage_type=S3
    cloud_native_storage.enable=true
    cloud_native_storage.storage_root_path=<s3://test-demo/starrocks/>
    aws.s3.use_instance_profile=true
    aws.s3.region=us-east-1
    aws.s3.endpoint=<https://s3.amazonaws.com>

starrocksCnSpec:
  # number of replicas to deploy for a CN statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/cn-ubuntu
    tag: ""
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  serviceAccountName: kubvir-starrocks-s3-sa
  config: |
    cloud_storage_type=S3
    cloud_native_storage.enable=true
    cloud_native_storage.storage_root_path=<s3://test-demo/starrocks/>
    aws.s3.use_instance_profile=true
    aws.s3.region=us-east-1
    aws.s3.endpoint=<https://s3.amazonaws.com>
    aws.s3.max_connections=64
Am I missing something?
a
Hi, Could you check 
show compute nodes;
command? You should verify whether a compute node is actually connected and active.
r
I got this working. Now I can see my s3 bucket in storage volumes but when I am trying to create the table , somehow it is taking my Node IAM role and not the Pod Service Account. Due to which getting this permission error:
Copy code
CREATE TABLE IF NOT EXISTS crashdata (
    ->     CRASH_DATE DATETIME,
    ->     BOROUGH STRING,
    ->     ZIP_CODE STRING,
    ->     LATITUDE INT,
    ->     LONGITUDE INT,
    ->     LOCATION STRING,
    ->     ON_STREET_NAME STRING,
    ->     CROSS_STREET_NAME STRING,
    ->     OFF_STREET_NAME STRING,
    ->     CONTRIBUTING_FACTOR_VEHICLE_1 STRING,
    ->     CONTRIBUTING_FACTOR_VEHICLE_2 STRING,
    ->     COLLISION_ID INT,
    ->     VEHICLE_TYPE_CODE_1 STRING,
    ->     VEHICLE_TYPE_CODE_2 STRING
    -> );
ERROR 1064 (HY000): fail to create tablet: 10001: [Internal error: starlet err [RequestID=959X5VTBKMG0FB5P][StatusCode=403]Put object <s3://test-demo/starrocks/405f4c59-ea08-4b96-872b-308b8ddabce4/db10130/10177/10179/SCHEMA_00000000000027C2> error: User: arn:aws:sts::1111111111:assumed-role/kubvir_dock_role/i-089cf543240a2801d is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::test-demo/starrocks/405f4c59-ea08-4b96-872b-308b8ddabce4/db10130/10177/10179/SCHEMA_00000000000027C2" because
This is my values.yaml
Copy code
prefix: kubvir
metrics:
  serviceMonitor:
    # Whether to expose metrics to Prometheus by ServiceMonitor.
    # Note: make sure the prometheus operator is installed in your cluster.
    # If prometheus is not installed by operator, you can add annotations on k8s service to expose metrics.
    # See <https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-prometheus-grafana.md#51-turn-on-the-prometheus-metrics-scrape-by-adding-annotations> for more details.
    enabled: true
starrocksCluster:
  # the namespace of starrockscluster cluster, if not set, the release namespace will be used.
  namespace: "dockprivileged"
  # specify the BE/CN deployment or not.
  enabledBe: false
  enabledCn: true
initPassword:
  enabled: true
  passwordSecret: starrocks-root-pass
#starrocksFeProxySpec:
#  enabled: true
#  service:
#      type: LoadBalancer
#      annotations:
#        <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: "nlb"
#        <http://service.beta.kubernetes.io/aws-load-balancer-scheme|service.beta.kubernetes.io/aws-load-balancer-scheme>: "internet-facing"
starrocksFESpec:
  service:
    type: ClusterIP
    ports:
      - name: query
        port: 9030
      - name: http
        port: 8030
  # number of replicas to deploy for a FE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/fe-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  serviceAccountName: kubvir-starrocks-s3-sa
  config: |
    run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = test-demo/starrocks/
    aws_s3_region = us-east-1
    aws_s3_endpoint=<https://s3.amazonaws.com>
    enable_load_volume_from_conf=true
    aws_s3_use_instance_profile=true

starrocksBeSpec:
  # number of replicas to deploy for a BE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/be-ubuntu
    tag: ""
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi

starrocksCnSpec:
  # number of replicas to deploy for a BE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/cn-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  serviceAccountName: kubvir-starrocks-s3-sa
a
hum... 🤔 can you check your iam role permission, and service account value setting. Please check whether the service account resource has been created correctly and whether it is properly injected into each pod.
Copy code
# add service account to all component
starrocksCluster:
  componentValues:
    serviceAccount: {input_your_sa}
Copy code
# If you need to deploy other resources, e.g., serviceAccount, you can add them here.
  # You can even deploy resources to different namespaces
  resources:
    - apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: {your_service_account_name}
        namespace: {your_namespace}
        annotations:
          eks.amazonaws.com/role-arn: {your_iam_arn}
r
Thanks @ahn heesuk for this info. Now atleast I can see in the pod that serviceAccount mapped with it is the one I created for this use case. Earlier it was taking default. But still error remains the same. It is still reading S3 from the Instance Profile role and not from this Service Account
a
@Rumit Jain Ah, maybe I think.. could you change your 
aws_s3_path
like bellow : • as-is : 
test-demo/starrocks/
• tobe : 
test-demo/starrocks
if your IRSA is valid, it can be trouble with 
aws_s3_path
👀
r
Got this working. Issue was with multiple wrong configurations. 1. FE and CN were getting deployed on different Nodes which were in different namespaces. 2. aws_s3_use_instance_profile set to true so this should be false
👍 1
Hey @ahn heesuk Meed you blessing again. I am trying now FE+BN setup and both the pods are up and running . But when I am inserting records in the table it is giving me this error:
Copy code
curl --location-trusted -u root             \
    -T ./NYPD_Crash_Data.csv                \
    -H "label:crashdata-0"                  \
    -H "column_separator:,"                 \
    -H "skip_header:1"                      \
    -H "enclose:\""                         \
    -H "max_filter_ratio:1"                 \
    -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \
    -XPUT <http://localhost:8030/api/quickstart/crashdata/_stream_load>
Enter host password for user 'root':
{
    "TxnId": 22,
    "Label": "crashdata-0",
    "Status": "Fail",
    "Message": "FE RPC failure, address=TNetworkAddress(hostname=kube-starrocks-fe-0.kube-starrocks-fe-search.dockprivileged.svc.cluster.local, port=9020), reason=TProtocolException: Invalid data",
    "NumberTotalRows": 0,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 0,
    "LoadTimeMs": 0,
    "BeginTxnTimeMs": 11,
    "StreamLoadPlanTimeMs": 0,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 0,
    "CommitAndPublishTimeMs": 0
}
I deleted pvc's, uninstalled installed the chart multiple times but still no luck
This is my values.yaml
Copy code
metrics:
  serviceMonitor:
    # Whether to expose metrics to Prometheus by ServiceMonitor.
    # Note: make sure the prometheus operator is installed in your cluster.
    # If prometheus is not installed by operator, you can add annotations on k8s service to expose metrics.
    # See <https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-prometheus-grafana.md#51-turn-on-the-prometheus-metrics-scrape-by-adding-annotations> for more details.
    enabled: true
starrocksCluster:
  # the namespace of starrockscluster cluster, if not set, the release namespace will be used.
  namespace: "dockprivileged"
  # specify the BE/CN deployment or not.
  enabledBe: true
  enabledCn: false
  componentValues:
    serviceAccount: srekubvir-starrocks-s3-sa
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]
#initPassword:
#  enabled: true
#  passwordSecret: starrocks-root-pass
starrocksFeProxySpec:
  enabled: true
  service:
      type: LoadBalancer
      #annotations:
      #  <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: "nlb"
      #  <http://service.beta.kubernetes.io/aws-load-balancer-scheme|service.beta.kubernetes.io/aws-load-balancer-scheme>: "internet-facing"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]
starrocksFESpec:
  service:
    type: ClusterIP
    ports:
      - name: query
        port: 9030
      - name: http
        port: 8030
  # number of replicas to deploy for a FE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/fe-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]

starrocksBeSpec:
  # number of replicas to deploy for a BE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/be-ubuntu
    tag: ""
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]
a
Are you testing it in a local environment right now, or is it already running in the cloud? 👀
r
@ahn heesuk in Cloud..AWS EKS
Also in fe.warn.log, I am seeing these WARNINGS :
Copy code
2025-11-25 16:34:52.122+08:00 WARN (heartbeat mgr|15) [HeartbeatMgr.runAfterCatalogReady():169] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.UnknownHostException: kube-starrocks-be-0.kube-starrocks-be-search.dockprivileged.svc.cluster.local
 2025-11-25 16:34:52.324+08:00 WARN (tablet checker|37) [TabletChecker.doCheckOnePartition():494] tablet: 10151 is in unhealthy state: VERSION_INCOMPLETE, but there are no healthy replicas, can not repair
 2025-11-25 16:34:57.346+08:00 WARN (UpdateBlacklistThread|110) [HostBlacklist.remove():105] remove black list: 10001, at: 2025-11-25T16:34:57.346765411
 2025-11-25 16:34:57.347+08:00 WARN (UpdateBlacklistThread|110) [HostBlacklist.refresh():230] remove nodeID 10001 from blacklist
 2025-11-25 16:35:32.859+08:00 WARN (thrift-server-pool-1|192) [LeaderImpl.finishTask():237] cannot find task. type: PUBLISH_VERSION, backendId: 10001, signature: 1002
Copy code
Starrocks > SHOW BACKENDS\G;
*************************** 1. row ***************************
            BackendId: 10001
                   IP: kube-starrocks-be-0.kube-starrocks-be-search.dockprivileged.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2025-11-25 16:34:57
        LastHeartbeat: 2025-11-25 16:38:12
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
            TabletNum: 113
     DataUsedCapacity: 66.444 KB
        AvailCapacity: 1016.252 GB
        TotalCapacity: 1023.500 GB
              UsedPct: 0.71 %
       MaxDiskUsedPct: 0.71 %
               ErrMsg: 
              Version: 4.0.1-cd9df36
               Status: {"lastSuccessReportTabletsTime":"2025-11-25 16:37:32"}
    DataTotalCapacity: 1016.252 GB
          DataUsedPct: 0.00 %
             CpuCores: 1
             MemLimit: 1.620GB
    NumRunningQueries: 0
           MemUsedPct: 12.24 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 0B/810GB, MemUsage: 2.6KB/331.7MB
             Location: 
           StatusCode: OK
1 row in set (0.00 sec)

ERROR: 
No query specified
BE pos is also healthy
a
That's strange. You're using the test from the StarRocks quickstart, right? Maybe the test dataset itself is broken. How about creating a small sample dataset and checking whether the input works properly?
r
Tried with simple dataset as well. Same error.
a
oh,
you set up shared nothing mode, right?
r
yes..only FE+BE
a
in your log, java.net.UnknownHostException error is occured,
r
strange...I did nothing and retried inserting the data. And it worked now
🤔 1
@ahn heesuk Can you guide what is wrong in this configuration. FE pods are not coming healthy and error logs are also not printing anything informative.
Copy code
metrics:
  serviceMonitor:
    # Whether to expose metrics to Prometheus by ServiceMonitor.
    # Note: make sure the prometheus operator is installed in your cluster.
    # If prometheus is not installed by operator, you can add annotations on k8s service to expose metrics.
    # See <https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-prometheus-grafana.md#51-turn-on-the-prometheus-metrics-scrape-by-adding-annotations> for more details.
    enabled: true
starrocksCluster:
  # the namespace of starrockscluster cluster, if not set, the release namespace will be used.
  # namespace: "{{ .Values.global.namespace }}"
  # specify the BE/CN deployment or not.
  enabledBe: false
  enabledCn: true
  componentValues:
    serviceAccount: kubvir-starrocks-s3-sa
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]
#initPassword:
#  enabled: true
#  passwordSecret: srekubvir-starrocks-root-password
#starrocksFeProxySpec:
#  enabled: true
#  service:
#      type: LoadBalancer
      #annotations:
      #  <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: "nlb"
      #  <http://service.beta.kubernetes.io/aws-load-balancer-scheme|service.beta.kubernetes.io/aws-load-balancer-scheme>: "internet-facing"
starrocksFESpec:
  service:
    type: ClusterIP
    ports:
      - name: query
        port: 9030
      - name: http
        port: 8030
  # number of replicas to deploy for a FE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/fe-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 1
      memory: 2Gi
  config: |
    run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = kubvir-starrocks/starrocks/
    aws_s3_region = us-east-1
    aws_s3_endpoint = <https://s3.amazonaws.com>
    enable_load_volume_from_conf = true
    aws_s3_use_instance_profile = false

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]

starrocksBeSpec:
  # number of replicas to deploy for a BE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/be-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]

starrocksCnSpec:
  # number of replicas to deploy for a BE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/cn-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi

  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]
As soon I add config| section in fe, pod do not come healthy.
a
How about starrocks operator log?
r
Yes @ahn heesuk there are these error logs in operator but no clue what is the error
Copy code
2025-11-26T10:52:29.675+0800	INFO	StarRocksClusterReconciler	begin to reconcile StarRocksCluster	{"name": "kube-starrocks", "namespace": "dockprivileged"}
2025-11-26T10:52:29.675+0800	INFO	StarRocksClusterReconciler	get StarRocksCluster CR from kubernetes	{"name": "kube-starrocks", "namespace": "dockprivileged"}
2025-11-26T10:52:29.675+0800	INFO	StarRocksClusterReconciler	sub controller sync spec	{"name": "kube-starrocks", "namespace": "dockprivileged", "subController": "feController"}
2025-11-26T10:52:29.675+0800	INFO	StarRocksClusterReconciler.feController	fetch configmap from kubernetes	{"name": "kube-starrocks", "namespace": "dockprivileged", "action": "SyncCluster", "name": "kube-starrocks-fe-cm"}
2025-11-26T10:52:29.675+0800	INFO	StarRocksClusterReconciler.feController	create or update statefulset	{"name": "kube-starrocks", "namespace": "dockprivileged", "action": "SyncCluster", "name": "kube-starrocks-fe"}
2025-11-26T10:52:29.972+0800	ERROR	StarRocksClusterReconciler.feController	deploy statefulset failed	{"name": "kube-starrocks", "namespace": "dockprivileged", "action": "SyncCluster", "error": "Operation cannot be fulfilled on statefulsets.apps \"kube-starrocks-fe\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/StarRocks/starrocks-kubernetes-operator/pkg/subcontrollers/fe.(*FeController).SyncCluster
	/go/src/app/pkg/subcontrollers/fe/fe_controller.go:130
github.com/StarRocks/starrocks-kubernetes-operator/pkg/controllers.(*StarRocksClusterReconciler).Reconcile
	/go/src/app/pkg/controllers/starrockscluster_controller.go:93
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2025-11-26T10:52:29.972+0800	ERROR	StarRocksClusterReconciler	sub controller reconciles spec failed	{"name": "kube-starrocks", "namespace": "dockprivileged", "subController": "feController", "error": "Operation cannot be fulfilled on statefulsets.apps \"kube-starrocks-fe\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/StarRocks/starrocks-kubernetes-operator/pkg/controllers.(*StarRocksClusterReconciler).Reconcile
	/go/src/app/pkg/controllers/starrockscluster_controller.go:94
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2025-11-26T10:52:29.980+0800	ERROR	Reconciler error	{"controller": "starrockscluster", "controllerGroup": "starrocks.com", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"kube-starrocks","namespace":"dockprivileged"}, "namespace": "dockprivileged", "name": "kube-starrocks", "reconcileID": "517c5c81-3fe3-42e7-976d-7c40b9ef1eb1", "error": "Operation cannot be fulfilled on statefulsets.apps \"kube-starrocks-fe\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2025-11-26T10:52:29.980+0800	INFO	StarRocksClusterReconciler	begin to reconcile StarRocksCluster	{"name": "kube-starrocks", "namespace": "dockprivileged"}
a
I think this error occurs because the operator tried to update the FE custom resource, but the resource version had already changed. The operator uses optimistic locking based on the resourceVersion, so when it detects a version mismatch, the update fails. It might also be caused by a cache inconsistency inside the operator related to the resource version. I’m considering whether restarting the operator could help, since that would reset its internal cache.
r
I uninstalled operator and reinstalled but no luck. What I am assuming is somehow the config section inside the fe spec is not correct. But not sure what is not correct. Becuase if I remove the config properties, both fe and cn comes healthy.
Copy code
starrocksFESpec:
  service:
    type: ClusterIP
    ports:
      - name: query
        port: 9030
      - name: http
        port: 8030
  # number of replicas to deploy for a FE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/fe-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 1
      memory: 2Gi
  config: |
    run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = test-demo/starrocks/
    aws_s3_region = us-east-1
    aws_s3_endpoint = <https://s3.amazonaws.com>
    enable_load_volume_from_conf = true
    aws_s3_use_instance_profile = false
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: namespace
            operator: In
            values: ["dock"]
@ahn heesuk Another thing I noticed is if I keep aws_s3_use_instance_profile = true, then also fe and cn are coming healthy but I want to make it use serviceAccount role so I keep aws_s3_use_instance_profile = false and then fe pod is not coming up.
I deployed another shell debug pod and checked the fe.log and it has these log line:
Copy code
2025-11-26 18:19:57.701+08:00 FATAL (stateChangeExecutor|89) [GlobalStateMgr.createBuiltinStorageVolume():2749] Invalid aws credential configuration.
Not sure what am I missing here
a
@Rumit Jain in my case, I just used 
aws_s3_use_aws_sdk_default_behavior = true
option in 
FESpec
Copy code
# Use AWS SDK default credentials (supports IRSA), if you want to use your own credentials, set this to false
aws_s3_use_aws_sdk_default_behavior = true
When 
aws_s3_use_aws_sdk_default_behavior = true
is enabled, StarRocks uses the AWS SDK’s default credential chaining, which means it will automatically use the Web Identity Token environment variables injected by IRSA.
r
yes..I forgot to tell you..after trying this the issue got resolved. 🙂
🙌 1
One quick thing..have you tried making Follower an Observer? I am referring this but it is not working https://docs.starrocks.io/docs/administration/management/Scale_up_down/#scale-fe-out
a
Oh, I didn’t know StarRocks had this feature. In your use case, do you have a lot of user-facing queries coming into StarRocks?
It looks like this feature can’t be used with the Operator, and is only available when deploying StarRocks manually at least according to what I saw in another comment.
r
what is "deploy StarRocks manually" ?
a
It means you are not using the StarRocks Operator and instead deploying the FE and CN node resources manually. In this case, since the Operator isn’t involved, the user should add or remove nodes by running the StarRocks commands directly.
r
Hi @ahn heesuk I am seeing this issue:
Copy code
create Pod kube-starrocks-fe-0 in StatefulSet kube-starrocks-fe failed error: pods "kube-starrocks-fe-0" is forbidden: violates PodSecurity "restricted:latest": seccompProfile (pod or container "fe" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
We have PSS enabled in our prod servers. But I am not seeing setting securityCOntext in the Cluster template. ANy idea how to set this?
Open in Slack
PreviousNext
Powered by