http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> can i use WIF for GCS permissions f...
# questions-and-troubleshooting
d
@Rocky can i use WIF for GCS permissions for reading from hive external tables?
r
Yes, you can use GKE Workload Identity (which is Google's recommended way to manage permissions for GKE workloads) for GCS permissions when reading from Hive external tables. However, "Workload Identity Federation" (WIF) for non-GCP environments is not explicitly documented as a standalone property in StarRocks catalogs. For GKE Workload Identity If your StarRocks cluster is running on GKE with Workload Identity enabled, you can configure your Hive catalog to use the pod's identity without using a JSON key file. To do this, set the following property in your 
CREATE EXTERNAL CATALOG
statement:
Copy code
sql
CREATE EXTERNAL CATALOG hive_catalog_gcs
PROPERTIES
(
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<metastore_ip>:9083",
    "gcp.gcs.use_compute_engine_service_account" = "true"
);
In GKE, StarRocks will use the metadata server (intercepted by Workload Identity) to fetch the credentials of the Google Service Account (GSA) bound to your Kubernetes Service Account (KSA). Supported GCS Authentication Methods According to the StarRocks documentation, there are three primary ways to authenticate to GCS for Hive catalogs: 1. VM-based (Metadata Server): Uses the service account attached to the Compute Engine VM or GKE node. * 
"gcp.gcs.use_compute_engine_service_account" = "true"
2. Service Account JSON Keys: Uses a service account's private key directly. * 
"gcp.gcs.service_account_email"
* 
"gcp.gcs.service_account_private_key_id"
* 
"gcp.gcs.service_account_private_key"
3. Impersonation: Allows a VM or one service account to impersonate another service account that has the actual data access. * 
"gcp.gcs.impersonation_service_account" = "<target_gsa_email>"
Note on Workload Identity Federation (WIF) If you are referring to Workload Identity Federation for accessing GCS from outside of Google Cloud (e.g., from AWS, Azure, or on-premises), StarRocks does not currently list a specific catalog property for a WIF credential configuration file (like 
gcp.wif_config
). * Tip: If you are in a non-GCP environment and require WIF, you might attempt to set the 
GOOGLE_APPLICATION_CREDENTIALS
environment variable on your BE nodes to point to your WIF configuration JSON, as StarRocks often leverages the underlying Google Cloud SDKs. However, this is not an officially documented catalog property. References • integrations/authenticate_to_gcs.md • administration/management/FE_configuration.md • deployment/shared_data/gcs.md
d
@Rocky i mean passing 
GOOGLE_APPLICATION_CREDENTIALS
for auth
r
Yes, you can use the 
GOOGLE_APPLICATION_CREDENTIALS
environment variable for authentication, but you must still tell StarRocks to use the "Default Credential" provider in your catalog configuration. How to use it To enable StarRocks to pick up the JSON file pointed to by your environment variable, follow these steps: 1. Set the Environment Variable: Ensure 
GOOGLE_APPLICATION_CREDENTIALS
is exported in the environment where your FE and BE (or CN) processes are running. * For Linux deployments, add 
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/key.json
to your 
fe/conf/fe_env.sh
and 
be/conf/be_env.sh
. * For Kubernetes (GKE), this is typically handled by Workload Identity or by mounting a Secret as a file and setting the env var in the Pod spec. 2. Configure the Catalog: Use the 
gcp.gcs.use_compute_engine_service_account
property. Despite the name, setting this to 
true
tells StarRocks to use Google's Application Default Credentials (ADC) chain, which prioritizes the file path in your environment variable before falling back to the metadata server.
Copy code
sql
CREATE EXTERNAL CATALOG hive_gcs
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<metastore_host>:9083",
    "gcp.gcs.use_compute_engine_service_account" = "true"
);
Why this works Under the hood, when 
"gcp.gcs.use_compute_engine_service_account" = "true"
is set, StarRocks initializes its GCS client using the standard Google Cloud SDK credential discovery. This discovery follows this order: 1. The file pointed to by 
GOOGLE_APPLICATION_CREDENTIALS
. 2. The metadata server (GCP VM service account or GKE Workload Identity). Alternative: Explicit Keys If you prefer not to manage environment variables or file mounts on every node, the officially documented method for using a Service Account JSON file is to pass the specific fields directly into the catalog properties:
Copy code
sql
CREATE EXTERNAL CATALOG hive_gcs
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<metastore_host>:9083",
    "gcp.gcs.service_account_email" = "<mailto:your-sa@project.iam.gserviceaccount.com|your-sa@project.iam.gserviceaccount.com>",
    "gcp.gcs.service_account_private_key_id" = "...",
    "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
);
References • integrations/authenticate_to_gcs.md • data_source/catalog/hive_catalog.md References • integrations/authenticate_to_gcs.md • data_source/catalog/iceberg/iceberg_catalog.md
d
@Rocky 
GOOGLE_APPLICATION_CREDENTIALS
for reading from hive catalog
r
Yes, you can use the 
GOOGLE_APPLICATION_CREDENTIALS
 environment variable to authenticate with GCS when reading from a Hive catalog. Although the Hive catalog documentation focuses on explicit keys, setting the environment variable is supported via the Application Default Credentials (ADC) mechanism. Steps to use 
GOOGLE_APPLICATION_CREDENTIALS
 1. Set the Environment Variable You must export the variable on all FE and BE nodes (or CN nodes) in your StarRocks cluster. The JSON key file must be accessible at the same path on all nodes. Add the following to your 
fe/conf/fe_env.sh
and `be/conf/be_env.sh`:
Copy code
bash
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/your/gcp-key.json"
Restart the FE and BE services after setting this. 2. Create the Catalog When creating your Hive catalog, set the property 
"gcp.gcs.use_compute_engine_service_account" = "true"
. Despite its name, this property instructs StarRocks to use the Google SDK's "Default" credential provider. This provider automatically checks the 
GOOGLE_APPLICATION_CREDENTIALS
environment variable first before falling back to the metadata server.
Copy code
sql
CREATE EXTERNAL CATALOG hive_gcs_adc
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<metastore_host>:9083",
    "gcp.gcs.use_compute_engine_service_account" = "true"
);
Important Considerations * Permissions: The service account in your JSON file must have 
Storage Object Viewer
(or higher) permissions on the GCS bucket. * Metastore Access: Ensure your Hive Metastore (HMS) is also configured to access GCS, as StarRocks depends on the HMS to provide the file paths. * Path Prefix: When querying, ensure your table locations or file paths use the 
gs://
protocol. Why this works StarRocks uses the Google Cloud C++ SDK on the BE and the Hadoop-GCS connector on the FE. Both libraries follow the Application Default Credentials (ADC) flow: 1. Check the 
GOOGLE_APPLICATION_CREDENTIALS
environment variable. 2. Check for a service account bound to the VM instance (Compute Engine/GKE). By setting 
"gcp.gcs.use_compute_engine_service_account" = "true"
, you enable this search chain. References • integrations/authenticate_to_gcs.md • data_source/catalog/hive_catalog.md References • integrations/authenticate_to_gcs.md • data_source/catalog/unified_catalog.md • data_source/catalog/deltalake_catalog.md • data_source/catalog/hive_catalog.md • data_source/catalog/iceberg/iceberg_catalog.md
Open in Slack
PreviousNext
Powered by