@Rocky I setup an Iceberg REST catalog and for all DDL it works fine , however when trying to query the data I get the following even though when I check the REST response it comes with vended credentials it seems Starrocks ignores it and doesn't use it. I had to grant temporary permission to the bucket to the starrocks service account and it worked, however expectation is to used the vended credentials rather than the GCP Workload Service Account ERROR 1064 (HY000): Failed to get status for file: gs://my-sample-bucket/taxi_demo/beam_taxi_summary/metadata/snap-3893430296183294740-1-588a7968-9003-4f95-bb6a-d4edc0fdbcdd.avro

The error you are encountering—where DDL works but queries fail with a file access error—typically indicates that the StarRocks Backend (BE) nodes are unable to authenticate with Google Cloud Storage (GCS) using the credentials they expect. Based on the StarRocks documentation and recent release notes, here is the situation regarding Iceberg REST Catalog vended credentials for GCS: 1. Version Requirement (StarRocks 4.0+) Support for vended credentials (temporary storage tokens provided by the REST catalog) for Google GCS and Azure ADLS was officially introduced in StarRocks 4.0. * If you are on StarRocks 3.x or earlier: The engine likely ignores the vended credentials in the REST response and falls back to the system-level GCP credentials (the Workload Service Account or VM Instance Service Account). This explains why your query only worked after you granted permissions to that specific service account. * If you are on StarRocks 4.0+: This feature is supported. You should ensure your catalog is configured to use them. 2. Required Catalog Property To ensure StarRocks uses the credentials provided by the REST catalog, you must have the following property set in your

CREATE EXTERNAL CATALOG

statement (though it defaults to

true

in version 4.0+):

sql
PROPERTIES (
  ...
  "iceberg.catalog.vended-credentials-enabled" = "true"
)

3. Verification & Troubleshooting If you are already on version 4.0 or later and it still fails: * Check

iceberg.catalog.security

: If your REST catalog requires specific authentication to vend those credentials, ensure you have set

"iceberg.catalog.security" = "oauth2"

or

"jwt"

as required by your catalog provider (e.g., Polaris, Tabular). * Storage Credentials: For GCS vended credentials in 4.0+, you should not need to provide

gcp.gcs.*

storage properties in the catalog definition. If they are present, they might be taking precedence. * Logs: Check the BE logs (

<http://be.INFO|be.INFO>

or

be.WARNING

) on the nodes where the query failed. Look for entries related to

GcsFileSystem

or

Iceberg

to see if there is a more descriptive error about why the vended token was rejected or not found. Summary If you are running a version earlier than 4.0, StarRocks does not yet support vending credentials for GCS via the Iceberg REST API. In this case, the GCP Workload Service Account is the intended and required method for authentication. To use vended credentials as expected, you will need to upgrade to StarRocks 4.0+. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_rest_security.md • integrations/authenticate_to_gcs.md

@Rocky I am using 4.0

SELECT @@version_comment;
+-------------------+
| @@version_comment |
+-------------------+
| 4.0.9-f647589     |
+-------------------+
1 row in set (0.00 sec)

and I already have the configuration property in the catalog "iceberg.catalog.vended-credentials-enabled" = "true",

Based on your StarRocks version (4.0.9) and configuration, the engine is technically equipped to handle vended credentials for GCS. However, the behavior you are describing—where StarRocks falls back to the GCP Workload Service Account (ADC) and ignores the vended tokens—is typically caused by one of the following configuration nuances: 1. Catalog Security Mode Requirement Even if you set

vended-credentials-enabled

to

true

, StarRocks often requires the catalog's security protocol to be explicitly defined (e.g.,

oauth2

or

jwt

) to correctly initialize the credential-aware communication flow. If your

iceberg.catalog.security

is set to

NONE

(the default), the engine may bypass the logic that extracts and applies the storage tokens from the REST response. Recommendation: Ensure your catalog is configured with a security protocol, even if the REST server doesn't strictly enforce it for metadata:

sql
PROPERTIES (
  "iceberg.catalog.type" = "rest",
  "iceberg.catalog.security" = "oauth2", -- or "jwt"
  "iceberg.catalog.vended-credentials-enabled" = "true"
)

2. Conflicting GCS Properties If you have any legacy

gcp.gcs.*

properties in your catalog DDL (such as

gcp.gcs.use_compute_engine_service_account

), these can sometimes take precedence over the vended credentials logic in the Backend (BE) nodes. * Action: Remove any

gcp.gcs.*

properties from the

CREATE EXTERNAL CATALOG

statement. According to the 4.0 documentation, for a REST catalog with vended credentials, you should not need to configure any storage-layer parameters. 3. REST Server Property Mapping StarRocks expects the vended credentials in the Iceberg REST response to follow standard key names. For GCS, this is usually provided in the

config

block of the

LoadTableResponse

. * If your REST catalog returns the token using a non-standard key (e.g., something other than

gcs.oauth2.token

or the standard OAuth2

token

field), StarRocks won't recognize it. * Verification: Check your BE nodes'

<http://be.INFO|be.INFO>

logs during a query. Look for messages related to

IcebergRESTCatalog

or

GcsFileSystem

. If StarRocks fails to find the token in the response, you might see a log entry indicating it is falling back to default credentials. 4. Known Issue: Token Passthrough in 4.0.x There have been reported cases in early 4.0.x releases where the

X-Iceberg-Access-Delegation: vended-credentials

header was not consistently passed in every request, or the BE failed to refresh the token for long-running scans. If the steps above do not resolve the issue: 1. Check if the REST Catalog logs show StarRocks sending the

X-Iceberg-Access-Delegation: vended-credentials

header. 2. Try setting

iceberg.catalog.vended-credentials-enabled

explicitly to

"true"

(as a string) if you haven't already. 3. As a temporary workaround for 4.0.9, you may need to continue using the Service Account until a specific patch for your REST provider (e.g., BigLake or Polaris) is confirmed in the 4.0.x branch. Authoritative Source: StarRocks Iceberg REST Security Docs and Iceberg Catalog Parameters. References • data_source/catalog/iceberg/iceberg_catalog.md • integrations/authenticate_to_gcs.md

@Rocky Ther are no gcs.* configuration, there is nothing about ICeberg/GCS on the cn.INFO the header is being sent and the response comes with a "storage_credentials" with the expected info: (be aware for security some fields have been redacted with ...")

"storage-credentials": [
        {
            "config": {
                "expiration-time": "1778553169283",
                "gcs.oauth2.refresh-credentials-enabled": "true",
                "gcs.oauth2.refresh-credentials-endpoint": "v1/projects/..../credentials",
                "gcs.oauth2.token": "ya29.dr.s.ASN...",
                "gcs.oauth2.token-expires-at": "1778553169283"
            },
            "prefix": "<gs://my-sample-bucket/taxi_demo/beam_taxi_summary/>"
        }
		]

Based on the details you provided, it appears that StarRocks 4.0.9 is successfully requesting vended credentials (sending the correct header) and your REST catalog is responding with the credentials. However, there is a mismatch in how the credentials are being returned versus how StarRocks expects to parse them. The Problem: "Storage Credentials" vs. "Config" Properties In the Iceberg REST specification, there are two ways vended credentials can be returned: 1. Legacy/Standard Config (Supported by StarRocks): The credentials (like

gcs.oauth2.token

) are returned directly within the

config

map of the

LoadTableResponse

. 2. Storage Credentials Capability (What your catalog is using): The credentials are returned in a separate

"storage-credentials"

array/object. This is a newer, more flexible part of the Iceberg REST spec designed for multi-storage and prefix-based scoping. Your response shows the credentials inside the

"storage-credentials"

block:

json
"storage-credentials": [
    {
        "config": { "gcs.oauth2.token": "..." },
        "prefix": "gs://..."
    }
]

StarRocks 4.0.x currently expects the credentials to be in the main

config

map of the table response. Because they are nested inside the

storage-credentials

array, the StarRocks Backend (BE) nodes fail to "see" the token. As a result, the BE falls back to the system's default credentials (the GCP Workload Service Account), which lacks the permissions you removed, leading to the

Failed to get status for file

error. Recommended Solutions 1. Adjust the REST Catalog Response (Preferred) If you have control over the Iceberg REST Catalog (e.g., if you are using Polaris, BigLake, or a custom server), check if you can configure it to return the storage tokens in the flat

config

map of the response rather than the

storage-credentials

block. Many catalogs support a "compatibility mode" or "legacy vended credentials" mode for engines that haven't yet implemented the full

storage-credentials

capability. 2. Verify

iceberg.catalog.security

Even in version 4.0, if

iceberg.catalog.security

is not set to

oauth2

or

jwt

, StarRocks might skip certain credential-processing logic. Ensure your catalog DDL includes:

sql
PROPERTIES (
  "iceberg.catalog.security" = "oauth2",
  "iceberg.catalog.vended-credentials-enabled" = "true"
)

3. Monitor for StarRocks Updates The support for the official Iceberg

storage-credentials

capability (the array-based response) is an active area of development in the 4.x branch. You may want to check the StarRocks GitHub Issues or reach out on the StarRocks Slack to see if a patch for this specific response format is available in a more recent 4.0.x or 4.1 release. Summary The vended credentials feature is active, but StarRocks 4.0.9 is looking for the token in the table's config map and is ignoring the storage-credentials array. Until StarRocks adds support for the

storage-credentials

capability, you will likely need to keep the Service Account permissions active or force the REST catalog to provide the token in the standard

config

block. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_rest_security.md