@Rocky which versions of apache polaris is supported for starrocks 3.5.16?

StarRocks 3.5.16 supports Apache Polaris as an external Iceberg REST catalog. While the official documentation does not mandate a single specific version of Polaris, the integration is based on the Iceberg REST Catalog API specification. Any version of Apache Polaris that adheres to this standard is compatible. Key Compatibility Details: * Verified Versions: Community and integration guides specifically mention Apache Polaris 1.1.0-incubating and 1.0.1-incubating as working well with the StarRocks 3.5.x branch. * Protocol: StarRocks uses the

rest

catalog type to connect to Polaris. * Authentication: Supports OAuth2 and vended credentials, which allow Polaris to manage storage permissions (like S3/ADLS/GCS tokens) and pass them to StarRocks for secure data access. Example Configuration: To connect StarRocks 3.5.16 to Apache Polaris, you can use a command like this:

sql
CREATE EXTERNAL CATALOG polaris_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "http://<polaris_host>:8181/api/catalog",
    "iceberg.catalog.warehouse" = "<your_polaris_warehouse>",
    "iceberg.catalog.security" = "oauth2",
    "iceberg.catalog.oauth2.credential" = "<client_id>:<client_secret>",
    "iceberg.catalog.oauth2.scope" = "PRINCIPAL_ROLE:ALL"
);

Notable Features in 3.5.x for Polaris: * Credential Vending: Automatically uses temporary cloud credentials provided by Polaris (

iceberg.catalog.vended-credentials-enabled

is

true

by default). * Nested Namespaces: StarRocks 3.5 supports multi-level namespaces (e.g.,

catalog.db.parent_ns.child_ns.table

), which is a common requirement when using Polaris. For the most up-to-date compatibility, you should generally use the latest stable release of Apache Polaris (currently in the 1.x line) as it aligns with the REST API features implemented in StarRocks 3.5. References • data_source/catalog/iceberg/iceberg_catalog.md

@Rocky so oath2 and idp should be configured in polaris so starrocks could connect? or polaris interla authentication and its rsa key pair is ok?

For StarRocks 3.5.16, Polaris's internal authentication is sufficient, and you do not need an external Identity Provider (IDP) like Okta or Keycloak unless your organization requires it. Here is the breakdown of how the authentication works between StarRocks and Polaris: 1. Internal Authentication is Supported Apache Polaris has a built-in Internal Authentication system and its own OAuth2 Token Broker. When you create a "Principal" in Polaris, it generates a Client ID and Client Secret. StarRocks uses these to authenticate directly with Polaris. 2. RSA Key Pair vs. OAuth2 Credentials * StarRocks Side: StarRocks connects to the Polaris REST catalog using the standard OAuth2 "Client Credentials" flow. It expects a

client_id

and

client_secret

. * Polaris Side: Polaris can use an RSA key pair internally to sign the tokens it issues (via its

rsa-key-pair

token broker configuration). This is an internal Polaris setting and is transparent to StarRocks. * Configuration: You do not provide an RSA key to StarRocks. Instead, you provide the credentials generated by Polaris for your principal:

sql
CREATE EXTERNAL CATALOG polaris_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "http://<polaris_host>:8181/api/catalog",
    "iceberg.catalog.security" = "oauth2",
    -- Format is client_id:client_secret
    "iceberg.catalog.oauth2.credential" = "your_client_id:your_client_secret",
    "iceberg.catalog.oauth2.scope" = "PRINCIPAL_ROLE:ALL"
);

3. When is an IDP required? You would only configure an external IDP in Polaris if: * You want to sync users from an existing corporate directory. * You want to use JWT-based pass-through authentication, where StarRocks passes the user's own login token (e.g., from an external OIDC provider) to Polaris for row/column level permission checks. Summary for StarRocks 3.5.16: * Is Polaris internal auth OK? Yes. * Is an external IDP required? No. * How to connect? Use

iceberg.catalog.security = "oauth2"

with the

client_id:client_secret

generated by Polaris. Note: Ensure you include

"iceberg.catalog.oauth2.scope" = 'PRINCIPAL_ROLE:ALL'

in your properties, as Polaris requires this scope to correctly map the principal to its roles. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_rest_security.md

@Rocky and what if I have multiple polaris realms

If you have multiple Polaris realms, you can connect to each of them by creating separate StarRocks external catalogs. In Apache Polaris, a realm is a top-level isolation unit (like a tenant). Because the realm is typically part of the connection context or the URL path, StarRocks needs to be told which realm to target for a specific catalog. How to specify the Realm in StarRocks StarRocks 3.5.x handles this using the standard Iceberg REST property for custom headers. You should use the property prefix

iceberg.catalog.header.

followed by the header name

Polaris-Realm

. Example for "Realm A":

sql
CREATE EXTERNAL CATALOG polaris_realm_a
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "http://<polaris_host>:8181/api/catalog",
    -- Specify the realm via the Polaris-Realm header
    "iceberg.catalog.header.Polaris-Realm" = "realm_a",
    "iceberg.catalog.security" = "oauth2",
    "iceberg.catalog.oauth2.credential" = "client_id_a:client_secret_a",
    "iceberg.catalog.warehouse" = "warehouse_a"
);

Example for "Realm B":

sql
CREATE EXTERNAL CATALOG polaris_realm_b
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "http://<polaris_host>:8181/api/catalog",
    "iceberg.catalog.header.Polaris-Realm" = "realm_b",
    "iceberg.catalog.security" = "oauth2",
    "iceberg.catalog.oauth2.credential" = "client_id_b:client_secret_b",
    "iceberg.catalog.warehouse" = "warehouse_b"
);

Key Considerations: * Logical Isolation: Each realm in Polaris is completely independent. You cannot "switch" realms within a single StarRocks catalog; you must define a catalog for each realm you want to access. * Standard Headers: StarRocks passes any property starting with

iceberg.catalog.header.

directly as an HTTP header to the REST endpoint. This is the recommended way to handle Polaris's multi-tenancy requirements. * Authentication: Remember that each realm has its own set of Principals and Credentials. Ensure the

iceberg.catalog.oauth2.credential

matches the client ID and secret generated within that specific Polaris realm. * Default Realm: If you do not specify a realm header, Polaris usually defaults to

default-realm

. Summary To handle multiple realms, create one StarRocks catalog per realm and use the

"iceberg.catalog.header.Polaris-Realm"

property to distinguish them. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_rest_security.md References • data_source/catalog/elasticsearch_catalog.md • data_source/catalog/maxcompute_catalog.md • data_source/catalog/hive_catalog.md

@Rocky should not I pass object storage credentials as well while creating external polaris catalog?

Generally, no, you do not need to pass static object storage credentials (like Access Key/Secret Key) when creating a Polaris catalog in StarRocks 3.5.16. This is because StarRocks supports a feature called Vended Credentials. How it Works When Vended Credentials are enabled, StarRocks does the following: 1. Authenticates with Polaris using the OAuth2 client credentials you provided. 2. Requests data from Polaris. 3. Receives temporary, short-lived storage credentials (e.g., S3 Session Tokens) directly from Polaris along with the table metadata. 4. Accesses the object storage (S3/GCS/Azure) using these temporary tokens. Recommended Configuration To use this, you should set

"iceberg.catalog.vended-credentials-enabled" = "true"

. While this is often the default behavior for REST catalogs in newer versions, explicitly setting it ensures StarRocks will look for credentials in the Polaris response.

sql
CREATE EXTERNAL CATALOG polaris_vended
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "rest",
    "iceberg.catalog.uri" = "http://<polaris_host>:8181/api/catalog",
    "iceberg.catalog.security" = "oauth2",
    "iceberg.catalog.oauth2.credential" = "<client_id>:<client_secret>",
    "iceberg.catalog.warehouse" = "<warehouse_name>",
    
    -- Tell StarRocks to use temporary credentials from Polaris
    "iceberg.catalog.vended-credentials-enabled" = "true",
    
    -- You may still need to specify the region for S3
    "aws.s3.region" = "us-east-1"
);

When WOULD you need storage credentials? You would only need to pass static credentials if: 1. Vended credentials are disabled in your Polaris server configuration. 2. Polaris does not have the necessary IAM roles to "vend" tokens for the underlying bucket. 3. You prefer to manage storage permissions entirely within StarRocks using a single high-privilege IAM user or Instance Profile. Why Vended Credentials are better: * Security: StarRocks never sees your master storage keys. If a StarRocks node is compromised, it only has access to temporary tokens. * Centralized Management: You manage all storage permissions (who can see which bucket) inside Polaris rather than duplicating that logic in StarRocks. Important: Ensure your Polaris "Storage Config" is correctly set up with the appropriate IAM roles, otherwise, Polaris will fail to generate the temporary tokens for StarRocks. References • data_source/catalog/iceberg/iceberg_rest_security.md • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/iceberg/iceberg_rest_s3.md • data_source/catalog/iceberg/iceberg_catalog.md References • data_source/catalog/iceberg/iceberg_rest_security.md • data_source/catalog/iceberg/iceberg_catalog.md