@Rocky Does Starrocks support credential vending for GCP Lakehouse for Iceberg (previously known as BigLake)?

Yes, StarRocks supports credential vending for GCP Lakehouse for Iceberg (BigLake) starting from version 4.0. Through its integration with the Iceberg REST Catalog protocol, StarRocks can use temporary, short-lived credentials provided by the BigLake service instead of requiring long-lived service account keys or direct bucket access. How it works In this "catalog-centric" security model, the authentication and authorization flow is as follows: 1. Authentication: StarRocks authenticates with the BigLake REST Catalog endpoint. 2. Credential Vending: Upon a query, the BigLake catalog "vends" short-lived GCS access tokens to StarRocks. 3. Data Access: StarRocks uses these temporary tokens to read or write the data files in GCS directly. Configuration To enable this, you must create an Iceberg catalog in StarRocks with the type set to

rest

and ensure credential vending is enabled (it is

true

by default in v4.0+).

sql
CREATE EXTERNAL CATALOG biglake_iceberg
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "<https://biglake.googleapis.com/iceberg/v1/restcatalog>",
    "iceberg.catalog.warehouse" = "<bq://projects/<project_id>/locations/<location>/catalogs/<catalog_id>>",
    "iceberg.catalog.security" = "google", -- Native Google authentication (available in v4.x)
    "iceberg.catalog.vended-credentials-enabled" = "true",
    "gcp.gcs.use_compute_engine_service_account" = "true" -- Use VM service account to talk to BigLake
);

Key Properties *

iceberg.catalog.vended-credentials-enabled

: When set to

true

, StarRocks will request and use the tokens provided by the REST catalog for storage access. *

iceberg.catalog.security

: Recent versions (v4.0+) added support for

google

as a security type to facilitate native authentication with BigLake using Google's

AuthManager

. *

iceberg.catalog.rest.header.X-Iceberg-Access-Delegation

: If needed for specific manual configurations, StarRocks can pass the

vended-credentials

header required by the BigLake spec. For earlier versions or manual token management, you can also use

iceberg.catalog.security = oauth2

and provide an explicit

iceberg.catalog.oauth2.token

. References • developers/code-style-guides/restful-api-standard.md • data_source/catalog/iceberg/iceberg_rest_security.md • data_source/catalog/iceberg/iceberg_catalog.md

@Rocky Thanks, but this doesn't match the source. There is no google value for iceberg.catalog.security. The enum in IcebergRESTCatalog.java is NONE/OAUTH2/JWT only, including on current main, and the FE only bundles iceberg-aws, not iceberg-gcp, so Iceberg's GoogleAuthManager isn't on the classpath either. Vended credentials also don't work against BigLake in practice: StarRocks only reads gcs.oauth2.token from the flat config map of the loadTable response, but BigLake returns credentials in the storage-credentials field, which StarRocks doesn't handle (and IcebergCachingFileIO doesn't implement SupportsStorageCredentials). We confirmed with a curl replay of loadTable that the flat map comes back empty. We got it working by adding the iceberg-gcp jar to the FE and using iceberg.catalog.rest.auth.type=GoogleAuthManager, with direct bucket IAM for data access but were wondering if there was out of the box support for it. Am I mistaken?

You are correct, and I apologize for the initial inaccuracy—I was conflating general vended credential support (which StarRocks has for AWS and Polaris) with native BigLake/GCP support, which is indeed more limited. Your technical assessment is spot on: 1. Missing

google

Security Type: The

iceberg.catalog.security

enum in the current standard release (and even on

main

as you've seen) is limited to

NONE

,

OAUTH2

, and

JWT

. There is no native

google

auth provider integrated into the StarRocks core REST catalog implementation yet. 2.

storage-credentials

Gap: While StarRocks supports "vended credentials" in a broad sense, it currently expects these credentials to be delivered in the

config

flat-map of the Iceberg REST

LoadTableResponse

(standard for AWS S3 and some other implementations). BigLake, however, returns these in the specific

storage-credentials

field defined in the Iceberg REST spec. 3. Missing GCP Bundles: The StarRocks FE does not currently bundle

iceberg-gcp

in its standard distribution, which is why

GoogleAuthManager

is not available on the classpath out-of-the-box. 4.

IcebergCachingFileIO

: Since this wrapper doesn't implement

SupportsStorageCredentials

, even if the REST response was parsed correctly, the IO layer wouldn't know how to apply those specific GCP-scoped tokens. Current Status There is an active effort (tracked in Issue #67726: Support Google AuthManager for Iceberg REST Catalog (BigLake)) to bridge this exact gap by adding native support for

iceberg.catalog.security = google

and ensuring the

storage-credentials

field is handled. The Recommendation Until that feature is officially merged and released, your current approach is the correct "power-user" workaround: * Jar Injection: Adding the

iceberg-gcp

bundle to the FE classpath. * Custom Auth: Using

iceberg.catalog.rest.auth.type = GoogleAuthManager

. * IAM: Relying on direct service account or workload identity permissions for the actual GCS data access. In summary, there is no out-of-the-box support for BigLake's specific

storage-credentials

vending flow today. The documentation's mention of vended credentials for v4.0 primarily refers to the AWS/OAuth2 patterns where tokens are passed via the standard config map. References • Issue #67726 - Support Google AuthManager for Iceberg REST Catalog (BigLake) • Issue #56817 - Add Vended Credentials mechanism for Azure (ADLS) (Related work on expanding vended credentials) • data_source/catalog/iceberg/iceberg_catalog.md References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_rest_security.md