I am a bit lost and need some helm urgently. I am trying to run starrocks shared everything cluster using helm charts on kubernetes. No matter what I cannot create any tables. All the tables FE is trying to create have the same problem and time out. In the FE:

2025-12-30 09:09:12.201+02:00 ERROR (TableKeeper|121) [TableKeeper.run():76] error happens in Keeper: com.starrocks.common.DdlException: Table creation timed out. unfinished replicas(1/1): 10083(yoav-starrocks-cn-0.yoav-starrocks-cn-search.default.svc.cluster.local)  timeout=10s

And in the CN:

W20251230 21:10:13.423729 140596591175232 agent_task.cpp:250] create table failed. status: Timeout: the task waits too long in the queue. timeout: 10000 ms, elapsed: 109000 ms, signature: 105190

The main issue is that no folder is being created in the S3 bucket. I found this line in the cn.WARNING log file:

W20251230 21:04:04.763927 140596591175232 agent_task.cpp:250] create table failed. status: Internal error: starlet err [RequestID=YJ32HVH4EF4RV29E][StatusCode=403]Put object <s3://data.warehouse.us-east-2.dev/starrocks/68040683-42a6-4799-9f7a-9b6d4dc44dee/db10004/104497/104499/SCHEMA_0000000000019832> error: Access Denied

Adding

aws_sdk_logging_trace_enabled = true

to the CN config yielded an empty log file. I added

aws_s3_use_aws_sdk_default_behavior=true

to the CN config and all it did was remove the

Access Denied

error message, but still, no folder was created in S3. Attached is the

values.yaml

file I am using.

nameOverride: "kube-starrocks-test-1"
initPassword:
  enabled: false
  isInstall: true
  password: ""
  passwordSecret: ""
timeZone: Asia/Jerusalem
datadog:
  log:
    enabled: false
metrics:
  serviceMonitor:
    enabled: false
starrocksCluster:
  name: "test-1-starrock-cluster"
  namespace: ""
  annotations: {}
  enabledBe: false
  enabledCn: true
  disasterRecovery:
  componentValues:
    image:
      tag: "4.0-latest"
    hostAliases: []
    runAsNonRoot: false
    schedulerName: ""
    serviceAccount: ""
    imagePullSecrets: []
    tolerations: []
    topologySpreadConstraints: []
    nodeSelector: {}
    affinity: {}
    podLabels: {}
starrocksFESpec:
  replicas: 1
  image:
    repository: starrocks/fe-ubuntu
    tag: ""
  imagePullPolicy: IfNotPresent
  entrypoint: {}
  annotations: {}
  runAsNonRoot: false
  readOnlyRootFilesystem: false
  capabilities: {}
  sysctls: []
  service:
    type: "ClusterIP"
    loadbalancerIP: ""
    annotations: {}
    labels: {}
    ports: []
    loadBalancerSourceRanges: []
  imagePullSecrets: []
  serviceAccount: ""
  nodeSelector: {}
  podLabels: {}
  hostAliases: []
  schedulerName: ""
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
  resources:
    requests:
      cpu: 4
      memory: 4Gi
    limits:
      cpu: 8
      memory: 8Gi
  storageSpec:
    name: ""
    storageClassName: ""
    storageSize: 10Gi
    storageMountPath: ""
    logStorageClassName: ""
    logStorageSize: 5Gi
    logMountPath: ""
  emptyDirs: []
  hostPaths: []
  persistentVolumeClaimRetentionPolicy:
  config: |
    run_mode = shared_data
    LOG_DIR = ${STARROCKS_HOME}/log
    DATE = "$(date +%Y%m%d-%H%M%S)"
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    mysql_service_nio_enabled = true
    sys_log_level = INFO
    cloud_native_storage_type = S3
    aws_s3_endpoint= <https://s3.us-east-2.amazonaws.com>
    aws_s3_path = data.warehouse.us-east-2.dev/starrocks-test-1
    aws_s3_region = us-east-2 
    aws_s3_access_key = XXX
    aws_s3_secret_key = YYY
    aws_s3_use_aws_sdk_default_behavior=true
    aws_s3_use_instance_profile = false
    enable_load_volume_from_conf= true
    min_graceful_exit_time_second = 25
    aws_sdk_logging_trace_enabled = true
  configyaml: {}
  secrets: []
  configMaps: []
  terminationGracePeriodSeconds: 120
  startupProbeFailureSeconds:
  livenessProbeFailureSeconds:
  readinessProbeFailureSeconds:
  minReadySeconds: 0
  podManagementPolicy: Parallel
  lifecycle: {}
  sidecars: []
  sidecarsMap: {}
  initContainers: []
  maxUnavailablePods:
  shareProcessNamespace:
starrocksCnSpec:
  replicas: 1
  image:
    repository: starrocks/cn-ubuntu
    tag: ""
  imagePullPolicy: IfNotPresent
  entrypoint: {}
  serviceAccount: ""
  annotations: {}
  runAsNonRoot: false
  readOnlyRootFilesystem: false
  capabilities: {}
  sysctls: []
  service:
    type: "ClusterIP"
    loadbalancerIP: ""
    annotations: {}
    labels: {}
    ports: []
    loadBalancerSourceRanges: []
  imagePullSecrets: []
  nodeSelector: {}
  podLabels: {}
  hostAliases: []
  schedulerName: ""
  affinity: {}
  tolerations: []
  topologySpreadConstraints: []
  autoScalingPolicy: {}
  resources:
    requests:
      cpu: 4
      memory: 8Gi
    limits:
      cpu: 8
      memory: 8Gi
  storageSpec:
    name: ""
    storageClassName: ""
    storageSize: 1Ti
    storageCount: 1
    storageMountPath: ""
    logStorageClassName: ""
    logStorageSize: 20Gi
    logMountPath: ""
    spillStorageClassName: ""
    spillStorageSize: 0Gi
    spillMountPath: ""
  emptyDirs: []
  hostPaths: []
  persistentVolumeClaimRetentionPolicy:
  config: |
    sys_log_level = INFO
    thrift_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
  configyaml: {}
  secrets: []
  configMaps: []
  terminationGracePeriodSeconds: 120
  startupProbeFailureSeconds:
  livenessProbeFailureSeconds:
  readinessProbeFailureSeconds:
  minReadySeconds: 0
  podManagementPolicy: Parallel
  lifecycle: {}
  sidecars: []
  sidecarsMap: {}
  initContainers: []
  maxUnavailablePods:
  shareProcessNamespace:
secrets: []
configMaps: []
resources: []
starrocksFeProxySpec:
  enabled: false

Have you bound the IAM roles to your instances? https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources/#preparations

Checking

If you were tweaking and made changes the AKS PV that was created might have been created with all bad references. I would try removing the whole deployment (remove also the namespace to get rid of the PVC) and try again. Then try creating a table to force the use of your S3 and try to see any error from the logs

Will do

Seems so, I run on GCP so not sure about your AWS values, but according to documentation they seem OK https://docs.starrocks.io/docs/deployment/shared_data/s3/#deploy-a-shared-data-starrocks-cluster

It seems this is a bug. In version

4.0.2

it works, and in version

4.0.3

it does not. I've seen off behavior like this also in the 3.5.x branch. https://github.com/StarRocks/starrocks/issues/67379