Hi Team, I'm trying to deploy a shared-data StarRo...
# questions-and-troubleshooting
r
Hi Team, I'm trying to deploy a shared-data StarRocks cluster on Kubernetes using the official Helm chart. According to the architecture docs (https://docs.starrocks.io/docs/introduction/Architecture/#shared-data), the shared-data mode “consists of only FE and CN nodes” and BEs are replaced by object storage. However, when I actually deploy only FE + CN (with S3 + IRSA configured): •
SHOW STORAGE VOLUMES;
returns empty • Creating a table fails with “Cluster has no available capacity” • Storage volumes do NOT auto-create, even with:
Copy code
enable_load_volume_from_conf=true
cloud_native_storage.enable=true
cloud_storage_type=S3
• Adding a
"storage_volume" = "S3VOL"
property still fails If I deploy a BE node, everything works immediately — storage volume appears and table creation succeeds. So my question is: Does the current shared-data implementation still require BE nodes to exist even though they don’t store local data? If yes, is FE+CN-only mode not yet supported in the Kubernetes operator, despite what the architecture docs describe?
k
the
run_mode
for the FE must be
shared_data
and can't be changed after FE started, if the configuration is configured wrong, shall delete the setup and redo the fresh installation.
r
Got it.. I was missing this:
Copy code
metrics:
  serviceMonitor:
    # Whether to expose metrics to Prometheus by ServiceMonitor.
    # Note: make sure the prometheus operator is installed in your cluster.
    # If prometheus is not installed by operator, you can add annotations on k8s service to expose metrics.
    # See <https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/integration/integration-prometheus-grafana.md#51-turn-on-the-prometheus-metrics-scrape-by-adding-annotations> for more details.
    enabled: true
starrocksCluster:
  # the namespace of starrockscluster cluster, if not set, the release namespace will be used.
  namespace: "dockprivileged"
  # specify the BE/CN deployment or not.
  enabledBe: false
  enabledCn: true
initPassword:
  enabled: true
  passwordSecret: starrocks-root-pass
starrocksFESpec:
  service:
    type: ClusterIP
    ports:
      - name: query
        port: 9030
      - name: http
        port: 8030
  # number of replicas to deploy for a FE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/fe-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  serviceAccountName: kubvir-starrocks-s3-sa
  config: |
    run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = test-demo/starrocks/
    aws_s3_region = us-east-1
    aws_s3_endpoint=<https://s3.amazonaws.com>
    enable_load_volume_from_conf=true
    aws_s3_use_instance_profile=true
    LOG_DIR = ${STARROCKS_HOME}/log
    DATE = "$(date +%Y%m%d-%H%M%S)"
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy"
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    mysql_service_nio_enabled = true
    sys_log_level = INFO

starrocksCnSpec:
  # number of replicas to deploy for a BE statefulset.
  replicas: 1
  image:
    # image sliced by "repository:tag"
    repository: starrocks/cn-ubuntu
    tag: "latest"
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited".
    limits:
      cpu: 1
      memory: 2Gi
  serviceAccountName: kubvir-starrocks-s3-sa
  config: |
      sys_log_level = INFO
      # ports for admin, web, heartbeat service
      thrift_port = 9060
      webserver_port = 8040
      heartbeat_service_port = 9050
      brpc_port = 8060
I created ServiceAccount and attached an IAM role with S3 required permission but now sure why I am seeing this error while creating the table:
Copy code
mysql> CREATE TABLE IF NOT EXISTS crashdata (     CRASH_DATE DATETIME,     BOROUGH STRING,     ZIP_CODE STRING,     LATITUDE INT,     LONGITUDE INT,     LOCATION STRING,     ON_STREET_NAME STRING,
CROSS_STREET_NAME STRING,     OFF_STREET_NAME STRING,     CONTRIBUTING_FACTOR_VEHICLE_1 STRING,     CONTRIBUTING_FACTOR_VEHICLE_2 STRING,     COLLISION_ID INT,     VEHICLE_TYPE_CODE_1 STRING,     VEHICLE_TYPE_CODE_2 STRING );
ERROR 1064 (HY000): fail to create tablet: 10001: [Internal error: starlet err [RequestID=NAEK53WF6DP6WE96][StatusCode=403]Put object <s3://test-demo/starrocks/06d7b9e3-0247-4830-a813-8a455c4e1d7e/db10925/16432/16434/SCHEMA_0000000000004031> error: User: arn:aws:sts::111111111111:assumed-role/kubvir_dock_role/i-068674bf5c4ad1eb3 is not authorized to perform: s3:PutObject on resource: "arn:aws:s3:::test-demo/starrocks/06d7b9e3-0247-4830-a813-8a455c4e1d7e/db10925/16432/16434/SCHEMA_0000000000004031" because
It is taking Node IAM role and not the pod Service Account. Am i missing something?
k
shall grant the IAM role and permission to the
service account
running starrocks cluster, and set the storage volume to use sdk default behavior.
r
Yes..already created IAM role in S3 and added all required permission for S3. Also created Service account and attached that IAM role with it. But somehow, while creating the table, it is complaining about the Node IAM role and not the ServiceAccount attached with the FE pod. Also, how to set storage volume to use sdk default behaviorr?