http://starrocks.io logo
Join Slack
Powered by
Hey <@U0A71G31CDV>, can you tell me what would be ...
# questions-and-troubleshooting
g
Hey @Rocky, can you tell me what would be the deployment procedure to have a Starrocks cluster in my K8S cluster, using a Storage Account from microsoft azure, and deploying using the Starrocks Operator. The basic steps would be (i suppose): • create a storage account in MS Azure • create an access (I'd rather it be a Workload identity if possible) • the install the starrocks cluster with the correct CRD? Can you detail all these steps ?
r
To deploy StarRocks on Kubernetes using Azure Storage and Workload Identity, follow these detailed steps. This procedure assumes you are using StarRocks v3.5.10 or later, which is the version where Azure Workload Identity support was officially introduced. 1. Azure Infrastructure Setup First, prepare the Azure resources and the trust relationship between your K8S cluster and Azure AD. * Create Storage Account: Create an Azure Data Lake Storage Gen2 (ADLS Gen2) account and a container (e.g., 
starrocks-container
). * Create User-Assigned Managed Identity: Create a Managed Identity (e.g., 
starrocks-identity
). * Assign Permissions: Grant the Managed Identity the Storage Blob Data Contributor role on your storage account/container. * Establish Federated Trust: 1. Retrieve the OIDC Issuer URL of your K8S cluster (e.g., 
az aks show --resource-group <rg> --name <cluster> --query "oidcIssuerProfile.issuerUrl" -o tsv
). 2. Create a Federated Identity Credential on the Managed Identity. Link it to the Kubernetes Namespace and ServiceAccount name you plan to use (e.g., namespace: 
starrocks
, service-account: 
starrocks-sa
). 2. Kubernetes Configuration Create a ServiceAccount that will be used by the StarRocks pods to assume the Azure identity.
Copy code
yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: starrocks-sa
  namespace: starrocks
  annotations:
    azure.workload.identity/client-id: "<YOUR_MANAGED_IDENTITY_CLIENT_ID>"
labels:
  azure.workload.identity/use: "true"
3. Deploy StarRocks Operator Install the StarRocks Operator and the Custom Resource Definition (CRD).
Copy code
bash
# 1. Install the CRD
kubectl apply -f <https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml>

# 2. Deploy the Operator
kubectl apply -f <https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml>
4. Deploy StarRocks Cluster (CRD) Configure the 
StarRocksCluster
resource. To use Azure Storage for your data, you must set the cluster to Shared-Data mode. In the example below, replace the placeholders with your Azure IDs:
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks
  namespace: starrocks
spec:
  starRocksFeSpec:
    replicas: 3
    serviceAccount: starrocks-sa  # Use the SA created in Step 2
    config: |
      run_mode = shared_data
      cloud_native_storage_type = ADLS2
      azure_adls2_endpoint = https://<ACCOUNT_NAME>.<http://dfs.core.windows.net|dfs.core.windows.net>
      azure_adls2_path = <CONTAINER_NAME>/<SUB_PATH>
      # Workload Identity Config (v3.5.10+)
      azure_adls2_oauth2_tenant_id = <AZURE_TENANT_ID>
      azure_adls2_oauth2_client_id = <MANAGED_IDENTITY_CLIENT_ID>
      azure_adls2_oauth2_token_file = /var/run/secrets/azure/tokens/azure-identity-token
    podLabels:
      azure.workload.identity/use: "true" # Required for webhook injection

  starRocksCnSpec:
    replicas: 3
    serviceAccount: starrocks-sa
    config: |
      # CNs also need access to the storage for caching/shuffling
      starlet_port = 9070
      storage_root_path = /opt/starrocks/cn/storage
    podLabels:
      azure.workload.identity/use: "true"
Key Considerations: * Version Compatibility: Ensure you use StarRocks v3.5.10+. If using an older version, Workload Identity is not supported, and you would need to use 
azure_adls2_shared_key
or 
azure_adls2_sas_token
in the config instead. * Shared-Data Mode: In this mode, StarRocks does not store data on local disks (except for cache); it stores all data in your Azure Storage Account. This is the standard "Cloud Native" deployment. * Webhook Injection: The 
azure.workload.identity/use: "true"
label is critical; it tells the Azure Workload Identity webhook to inject the identity token file into the pod at the path specified in 
azure_adls2_oauth2_token_file
.
* BE vs CN: For Kubernetes deployments, it is highly recommended to use CN (Compute Nodes) instead of BE (Backend) when running in 
shared_data
mode, as CNs are stateless and easier to scale. References • deployment/shared_data/azure.md
g
@Rocky If I want to use a single storage account, but for multiple clusters, is it possible ? For instance have a specific prefix for each cluster ( 
cluster1/
for cluster1 files, etc) ?
r
Yes, it is absolutely possible to use a single Azure Storage Account for multiple StarRocks clusters. You can isolate each cluster by specifying a unique sub-path (prefix) within the same container. How to configure isolation In your StarRocks FE configuration (within the 
StarRocksCluster
CRD), you use the 
azure_adls2_path
parameter. This parameter follows the format 
<container>/<directory>
. To isolate your clusters, simply provide a different directory name for each one: * Cluster 1 Configuration:
Copy code
yaml
    azure_adls2_endpoint = <https://mystorageaccount.dfs.core.windows.net>
    azure_adls2_path = starrocks-container/cluster1
* Cluster 2 Configuration:
Copy code
yaml
    azure_adls2_endpoint = <https://mystorageaccount.dfs.core.windows.net>
    azure_adls2_path = starrocks-container/cluster2
Important Considerations: 1. Unique Metadata: Each shared-data cluster manages its own metadata and data files. By pointing them to different sub-paths, they will remain completely independent and will not interfere with each other's data. 2. Shared Resources/Throttling: While the data is isolated logically, remember that both clusters will share the IOPS and throughput limits of the single Azure Storage Account. If you have multiple high-concurrency clusters, monitor the storage account metrics for throttling. 3. Workload Identity Permissions: Your Managed Identity needs 
Storage Blob Data Contributor
permissions on the storage account. If you use a single identity for both clusters, it will have access to both prefixes. If you need stricter security isolation, you could use two different Managed Identities and restrict their access to specific directories using Azure RBAC (though assigning roles at the directory level in ADLS Gen2 is typically done via ACLs rather than standard Azure RBAC). 4. Cleaning Up: If you delete a cluster, it will not automatically delete the files in the Azure Storage Account. You will need to manually remove the 
cluster1/
or 
cluster2/
directory if you want to reclaim space. References • deployment/shared_data/feature-support-shared-data.md • integrations/authenticate_to_azure_storage.md • administration/management/FE_configuration.md
g
@Rocky Should I use the 
enable_load_volume_from_conf
in the FE ? So that the volume is directly created? Also, will this volume be automatically used when I create databases?
r
Yes, you should use 
enable_load_volume_from_conf = true
if you want a seamless deployment where the storage account defined in your configuration is automatically used as the primary storage for your databases. Here is how it works and why you would use it: 1. Does it create the volume directly? Yes. When 
enable_load_volume_from_conf
is set to 
true
, StarRocks FE will automatically create a storage volume named 
builtin_storage_volume
during its first startup. It uses the 
azure_adls2_path
, 
azure_adls2_endpoint
, and credential settings (Workload Identity) you provided in the configuration file. * Note: From StarRocks v3.4.1 onwards, the default value for this parameter is 
false
. Therefore, you must explicitly set it to 
true
in your FE configuration if you want this automated behavior. 2. Is it automatically used for new databases? Yes. When this built-in volume is created, StarRocks automatically sets it as the Default Storage Volume for the cluster. In Shared-Data mode, every database and table needs a storage volume to know where to store data. Since this volume is the default, any 
CREATE DATABASE
or 
CREATE TABLE
statement you run will automatically use this Azure storage path without you needing to specify a 
STORAGE VOLUME
clause in your SQL. Summary of the Setup In your 
StarRocksCluster
CRD, ensure your FE config includes:
Copy code
yaml
spec:
  starRocksFeSpec:
    config: |
      run_mode = shared_data
      enable_load_volume_from_conf = true  # <--- Add this
      cloud_native_storage_type = ADLS2
      azure_adls2_endpoint = https://<ACCOUNT>.<http://dfs.core.windows.net|dfs.core.windows.net>
      azure_adls2_path = <CONTAINER>/<PREFIX>
      # ... rest of your Workload Identity config
Recommendation * Use 
true
 for your first cluster or if you want the "standard" Kubernetes experience where the infrastructure setup is handled by the initial configuration. * Use 
false
 only if you prefer to manually manage storage volumes via SQL (using 
CREATE STORAGE VOLUME
) after the cluster is up, which gives you more control over the volume name but requires an extra manual step before you can create any data. References • administration/management/FE_configuration.md • deployment/shared_data/azure.md References • administration/management/FE_configuration.md • deployment/shared_data/s3.md • deployment/shared_data/gcs.md • deployment/shared_data/minio.md
2 Views
Open in Slack
PreviousNext
Powered by