@Rocky I want to use an Azure storage account for starrocks, and I have 2 options: AZBLOB or ADLS2. Although I want also that the storage be replicated and handles versioning, and connect with Workload Identity. From the doc I understand that I have to use ADLS2, but I have these errors in the CNs:

W20260610 112253.585300 139407555728960 vacuum.cpp:139] Fail to delete: Internal error: starlet err Delete adls2 file log/0000000000003B1F_0000000000000841.log error: 409 This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.

What can I do ?

The error you're seeing (

409 This endpoint does not support BlobStorageEvents or SoftDelete

) is a known issue when using the Azure Data Lake Storage Gen2 (ADLS2) DFS endpoint with certain storage account features enabled. 1. Fix for the 409 Error The StarRocks vacuum process (Starlet) uses an internal driver for ADLS2 that is incompatible with Soft Delete and Blob Storage Events. * What to do: You must disable Soft Delete (for blobs and containers) and Blob Storage Events on the Azure Storage account. * Why: When these features are enabled, Azure rejects the specific metadata and deletion calls made via the

.<http://dfs.core.windows.net|dfs.core.windows.net>

endpoint. StarRocks manages its own data lifecycle (via the Vacuum process and FE trash bin), so external versioning or soft-deleting at the storage layer typically causes conflicts and prevents StarRocks from cleaning up old data files. 2. Choosing between AZBLOB and ADLS2 For StarRocks shared-data mode, the choice depends on your Hierarchical Namespace (HNS) setting: * ADLS2: Use this if HNS is Enabled. * AZBLOB: Use this if HNS is Disabled. _ _Note:* StarRocks documentation explicitly states that for

AZBLOB

(Azure Blob Storage) in shared-data mode, the hierarchical namespace must be disabled. 3. Connection with Workload Identity StarRocks supports Azure Workload Identity (federated identity for AKS), but it requires specific configuration: 1. Enable Native SDK: Ensure your FE configuration (

fe.conf

) has

azure_use_native_sdk = true

(available from v3.4.4). This allows the use of the Azure C++ SDK for authentication. 2. Authentication Properties: When creating your

STORAGE VOLUME

, use the following properties for Workload Identity:

sql
    CREATE STORAGE VOLUME azure_volume
    TYPE = ADLS2
    LOCATIONS = ("adls2://<container>@<account>.<http://dfs.core.windows.net/%22|dfs.core.windows.net/">)
    PROPERTIES (
        "enabled" = "true",
        "azure.adls2.endpoint" = "https://<account>.<http://dfs.core.windows.net|dfs.core.windows.net>",
        "azure.adls2.oauth2_use_managed_identity" = "true",
        "azure.adls2.oauth2_tenant_id" = "<tenant_id>",
        "azure.adls2.oauth2_client_id" = "<client_id_of_the_identity>"
    );

_Note: Even though the property says

managed_identity

, the underlying SDK uses the Workload Identity (federated token) when running in an environment where the

AZURE_FEDERATED_TOKEN_FILE

is present._ 4. Replication and Versioning * Replication: You can safely use Azure's replication options (LRS, GRS, or ZRS). These are handled at the infrastructure level and do not interfere with StarRocks' operations. * Versioning: As mentioned above, disabling storage-level versioning/soft-delete is required to avoid the 409 errors and ensure the Vacuum process can physically delete files when they are no longer needed by the database. Summary of Action: Go to your Azure Storage Account settings and disable Blob Soft Delete, Container Soft Delete, and ensure no Blob Storage Events are active. Then, restart your CNs or wait for the next Vacuum cycle to retry. References • introduction/versioning.md • deployment/shared_data/azure.md • deployment/upgrade.md

@Rocky Got it, but it is possible to perform point in time backups of the shared data? I expected to be able to handle this through versioning. What is the story about data restoration?

In StarRocks shared-data mode, data restoration and point-in-time recovery (PITR) are handled differently than in the traditional shared-nothing architecture. Because the storage is external (ADLS2/S3), the "story" focuses on Metadata Snapshots rather than physical file copying. 1. Why you can't use Azure Storage Versioning As you've seen with the 409 error, StarRocks' internal storage driver (Starlet) is incompatible with Azure's Soft Delete and Versioning features. * The Conflict: StarRocks manages its own data lifecycle. It uses a "Vacuum" process to identify and delete files that are no longer referenced by the metadata. If Azure "versions" or "soft-deletes" these files instead of physically removing them, the metadata and storage layer get out of sync, and the deletion calls fail. * The Recommendation: Disable these features at the Azure level and use StarRocks' built-in mechanisms for data protection. 2. How to achieve Point-in-Time Recovery (PITR) Instead of storage-level versioning, you should use Cluster Snapshots, a feature introduced specifically for shared-data clusters (Beta in v3.4.2, GA in v3.5). * What it does: It captures a consistent snapshot of the cluster's metadata (databases, tables, users, permissions) at a specific point in time. * How it works: Since the data files themselves are already immutable and stored on ADLS2, a snapshot only needs to record which files belonged to the cluster at that moment. * Restoration: To restore to a point in time, you can point a cluster to a previous snapshot. The recovery is near-instant because no data files need to be moved or copied—the cluster simply reloads the metadata associated with that snapshot. 3. Handling Accidental Deletions (The "Trash") For simple "oops" moments (like

DROP TABLE

), StarRocks has a built-in Trash mechanism: * When you delete a table or partition, it isn't physically deleted from ADLS2 immediately. * It is moved to a metadata "trash" for a configurable period (default is 1 day, controlled by

catalog_trash_expire_second

in FE). * You can use the

RECOVER

command to instantly restore the deleted object:

sql
    RECOVER TABLE my_table;

4. Important:

BACKUP

vs.

Cluster Snapshot

It is a common point of confusion: * The standard

BACKUP

and

RESTORE

commands (which export data to a separate repository) are not supported for shared-data clusters. * The Cluster Snapshot is the primary disaster recovery tool for shared-data mode. Summary of Data Protection Strategy: 1. Storage Level: Disable Soft Delete/Versioning to fix your 409 errors. Use Azure's LRS/GRS replication for hardware-level durability. 2. Accidental Deletion: Rely on the

RECOVER

command (Trash mechanism). 3. PITR / Disaster Recovery: Upgrade to v3.5.x and configure Cluster Snapshots. This allows you to roll back the entire cluster state to a known good point in object storage without the overhead of storage-level versioning. References • deployment/shared_data/feature-support-shared-data.md • administration/management/Backup_and_restore.md • faq/Exporting_faq.md • sql-reference/sql-statements/backup_restore/BACKUP.md