http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> In the context of Kubernetes hosted...
# questions-and-troubleshooting
s
@Rocky In the context of Kubernetes hosted starrocks db cluster and client applications, are there ways for client apps to authenticate against StarRocks cluster without managing shared secrets ?
r
In the context of Kubernetes-hosted StarRocks clusters and client applications, the most effective way to authenticate without managing shared secrets (like passwords) is by using JSON Web Token (JWT) Authentication, a feature introduced in StarRocks v3.5.0. By leveraging the Kubernetes ServiceAccount identity, your client applications can authenticate to StarRocks using a short-lived, automatically rotated token provided by the Kubernetes API server. How it works 1. Trust Establishment: StarRocks Frontends (FEs) are configured to trust the Kubernetes cluster as an Identity Provider (IdP) by pointing to the Kubernetes OpenID Connect (OIDC) discovery endpoint. 2. Identity Mounting: The client application (Pod) mounts its own identity token as a file using a Kubernetes Projected Volume. 3. Authentication: When the client connects to StarRocks, it sends this JWT instead of a password. StarRocks validates the token against the Kubernetes public keys and maps the token's identity to a user. ────────── Step-by-Step Implementation 1. Configure StarRocks to trust Kubernetes Create a Security Integration in StarRocks. This allows StarRocks to fetch the public keys (JWKS) from your Kubernetes API server to verify tokens.
Copy code
sql
CREATE SECURITY INTEGRATION k8s_auth
PROPERTIES (
    "type" = "authentication_jwt",
    -- Point to your K8s OIDC endpoint (accessible from FE pods)
    "jwks_url" = "<https://kubernetes.default.svc/openid/v1/jwks>",
    -- The field in the JWT used as the username
    "principal_field" = "sub", 
    -- The issuer URL of your K8s cluster (can be found via 'kubectl get --raw /.well-known/openid-configuration')
    "required_issuer" = "<https://kubernetes.default.svc.cluster.local>"
);
Note: Ensure your FE pods can reach 
kubernetes.default.svc
and trust its CA certificate (usually found at 
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
). 2. Configure the Client Application Pod Use a Projected ServiceAccount Token in your application's deployment. This ensures the token has a specific audience (e.g., 
starrocks
) and is refreshed automatically.
Copy code
yaml
apiVersion: v1
kind: Pod
metadata:
  name: client-app
spec:
  serviceAccountName: my-app-sa
  containers:
  - name: app
    image: my-app-image
    volumeMounts:
    - name: sa-token
      mountPath: /var/run/secrets/starrocks
  volumes:
  - name: sa-token
    projected:
      sources:
      - serviceAccountToken:
          path: token
          expirationSeconds: 3600
          audience: starrocks  # Matches StarRocks configuration
3. Connect to StarRocks The client application reads the token from 
/var/run/secrets/starrocks/token
and uses it to connect. * MySQL Client (v9.2+):
Copy code
bash
    mysql -h <fe_host> -P 9030 -u "system:serviceaccount:default:my-app-sa" \
      --authentication-openid-connect-client-id-token-file=/var/run/secrets/starrocks/token
* JDBC: For JDBC-based applications, you can use the StarRocks JDBC OAuth2 plugin which allows passing the token via connection properties or a custom authentication plugin. Key Advantages * Zero Secret Management: No passwords or API keys are stored in environment variables or Kubernetes Secrets. * Identity-Based: Permissions in StarRocks are tied directly to the Kubernetes ServiceAccount name. * Automated Rotation: Kubernetes manages the lifecycle and rotation of the tokens. * Virtual Users: With 
SECURITY INTEGRATION
, you don't need to manually run 
CREATE USER
for every service account; StarRocks can create temporary session-based users automatically. Alternative: Mutual TLS (mTLS) While StarRocks supports SSL/TLS for transport encryption (starting from v3.4.1), it is primarily used for securing the connection and server verification. For secret-less authentication, the JWT/OIDC method is the standard and most native approach for Kubernetes environments. References
s
@Rocky Our production environment is somewhat special in that clients applications can run inside a K8S cluster that is not the same as where StarRocks runs on. Can we still get this to work cross K8S cluster ?
r
Yes, you can absolutely achieve this cross-cluster. In fact, this is one of the primary use cases for the Security Integration feature introduced in StarRocks v3.5.0. Since StarRocks validates the token's signature using the public keys (JWKS) of the issuer, the physical location of the client application does not matter as long as the StarRocks Frontend (FE) can reach the client cluster's OIDC discovery endpoint to fetch those keys. How to set it up for Cross-Cluster Auth To allow applications in "Cluster B" to authenticate to StarRocks in "Cluster A" without secrets, follow these steps: 1. Obtain the OIDC Endpoint for Cluster B Kubernetes clusters (especially managed ones like EKS, GKE, or AKS) expose a public OIDC discovery endpoint. You can find this by running the following in the context of the client cluster:
Copy code
bash
kubectl get --raw /.well-known/openid-configuration | jq -r '.issuer'
The JWKS URL is typically the issuer URL followed by 
/openid/v1/jwks
. 2. Create the Security Integration in StarRocks (Cluster A) Run this command in StarRocks to trust the ServiceAccount tokens coming from the external cluster:
Copy code
sql
CREATE SECURITY INTEGRATION cluster_b_auth
PROPERTIES (
    "type" = "authentication_jwt",
    -- The JWKS URL of the CLIENT cluster (Cluster B)
    "jwks_url" = "https://<cluster-b-oidc-endpoint>/openid/v1/jwks",
    "principal_field" = "sub", 
    -- The Issuer URL of Cluster B
    "required_issuer" = "<https://kubernetes.default.svc.cluster.local>", 
    "required_audience" = "starrocks"
);
3. Enable the Authentication Chain Tell StarRocks to include this new integration in its login process:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("authentication_chain" = "cluster_b_auth");
4. Configure Client Pods in Cluster B Your client applications in the other cluster must mount their identity token with the correct audience to match your StarRocks configuration.
Copy code
yaml
# Deployment in Cluster B
spec:
  template:
    spec:
      containers:
      - name: client-app
        volumeMounts:
        - name: sr-token
          mountPath: /var/run/secrets/starrocks
      volumes:
      - name: sr-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              audience: starrocks  # Must match 'required_audience' in StarRocks
              expirationSeconds: 3600
Critical Requirements for Cross-Cluster 1. Network Connectivity: The StarRocks FE pods in Cluster A must be able to reach the 
jwks_url
of Cluster B. * If Cluster B is a managed cloud cluster (EKS/AKS/GKE), this is usually a public URL. * If Cluster B is private, you must ensure Cluster A's network can route to Cluster B's API server or a proxy that serves the discovery files. 2. CA Certificates: If Cluster B uses a private CA for its API server, you must mount that CA certificate into the StarRocks FE pods so they can securely fetch the JWKS. 3. Identity Mapping: When the client connects, their username in StarRocks will match the 
sub
claim of the Kubernetes token, which usually looks like: 
system:serviceaccount:<namespace>:<serviceaccount-name>
. Benefits of this Approach * Multiple Clusters: You can create multiple 
SECURITY INTEGRATION
objects, one for each client cluster you want to trust. * Dynamic Users: You don't need to manually 
CREATE USER
for every application. If the integration is valid, StarRocks creates a "virtual user" for that session based on the token. * No Secret Syncing: You never have to copy passwords or API keys between Cluster B and Cluster A. References • administration/user_privs/authentication/security_integration.md • best_practices/authentication_authorization.md
s
@Rocky With these virtual users, how could you setup associated permission, for example to prevent them from INSERT'ing into table.
r
To manage permissions for "virtual users" (session-scoped users) without manually managing passwords, you have two primary paths in StarRocks. Since these users aren't stored in the database's user table, you assign permissions by mapping their external identity (from the JWT) to StarRocks Roles. Option 1: Native User Mapping (Recommended for Specific Apps) Even though you are using JWT for authentication, you can still create a "shell" user in StarRocks. This user has no password; it simply acts as an anchor for permissions. 1. Create the user matching the Kubernetes 
sub
claim:
Copy code
sql
    -- The username must match the 'principal_field' (usually the 'sub' claim)
    CREATE USER `system:serviceaccount:default:my-app-sa`
    IDENTIFIED WITH k8s_auth; -- 'k8s_auth' is the name of your Security Integration
2. Grant limited permissions:
Copy code
sql
    -- Grant read-only access to a specific database
    GRANT SELECT ON ALL TABLES IN DATABASE reporting_db
    TO USER `system:serviceaccount:default:my-app-sa`;

    -- Do NOT grant INSERT, UPDATE, or DELETE
When the app logs in with its JWT, StarRocks matches the token to this user and applies the pre-defined permissions. ────────── Option 2: External Group Mapping (Scalable for many apps) If you have many applications and don't want to create every user manually, you can use External Groups. This requires a 
GROUP PROVIDER
to be configured alongside your Security Integration. 1. Define a Group Provider: You can configure StarRocks to extract "groups" from a specific claim in the JWT (e.g., if your OIDC provider adds a 
groups
or 
roles
claim). 2. Grant Roles to the External Group: Instead of granting to a user, you grant to the group that the application belongs to.
Copy code
sql
    -- Create a read-only role
    CREATE ROLE read_only_role;
    GRANT SELECT ON ALL TABLES IN DATABASE prod_db TO ROLE read_only_role;

    -- Map the external group (from the JWT) to this role
    GRANT ROLE read_only_role TO EXTERNAL GROUP "k8s-reader-apps";
3. Result: Any application presenting a JWT with the 
groups: ["k8s-reader-apps"]
claim will automatically inherit the 
read_only_role
for that session and will be unable to 
INSERT
. ────────── Option 3: Apache Ranger (Centralized) If your organization already uses Apache Ranger, StarRocks can delegate all authorization to it. _ When the client connects via JWT, StarRocks authenticates the user and then asks Ranger: _"Can user 
system:serviceaccount:default:my-app-sa
perform 
INSERT
on 
table_a
?"* * Ranger checks its policies and returns 
DENY
. Summary: How to prevent 
INSERT
? The most reliable way is to ensure that the Role assigned to the virtual user (either via the shell user in Option 1 or the group in Option 2) only contains 
SELECT
privileges. By default, in StarRocks, any user who is not granted the 
INSERT
privilege cannot perform it. Virtual users start with zero permissions (aside from those in the 
public
role), so you simply omit the 
INSERT
grant when setting them up. References • administration/user_privs/authentication/security_integration.md • administration/user_privs/authorization/privileges.md • best_practices/authentication_authorization.md • administration/user_privs/group_provider.md References • best_practices/authentication_authorization.md
s
@Rocky The JWT 
sub
field format is 
system:serviceaccount:<k8s-namespace>:<service-account-name>
. Is there a way to authenticate remote apps based only on the 
k8s-namespace
part ignoring that application service account name ?
r
Yes, you can authenticate applications based solely on their Kubernetes namespace by changing which field StarRocks uses to identify the "user." While the 
sub
claim contains the full string (
system:serviceaccount:<namespace>:<name>
), Kubernetes-issued JWTs also include a specific, separate claim that contains only the namespace name. The Solution: Change the 
principal_field
 Instead of using the default 
sub
field, you can configure StarRocks to look at the 
<http://kubernetes.io/serviceaccount/namespace|kubernetes.io/serviceaccount/namespace>
claim. 1. Create the Security Integration When you create the security integration, set the 
principal_field
to the Kubernetes namespace claim.
Copy code
sql
CREATE SECURITY INTEGRATION k8s_namespace_auth
PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "https://<cluster-b-oidc-endpoint>/openid/v1/jwks",
    -- Point StarRocks to the namespace claim instead of 'sub'
    "principal_field" = "<http://kubernetes.io/serviceaccount/namespace|kubernetes.io/serviceaccount/namespace>",
    "required_issuer" = "<https://kubernetes.default.svc.cluster.local>",
    "required_audience" = "starrocks"
);
2. Manage Permissions By using the namespace as the principal, all service accounts within the same namespace will appear to StarRocks as the same user (the name of the namespace). * Option A: Manual User (Native RBAC) If your namespace is named 
analytics-prod
, you create a single user in StarRocks with that name. Every application in that namespace will then share the same permissions.
Copy code
sql
    CREATE USER `analytics-prod` IDENTIFIED WITH authentication_jwt;

    -- Grant read-only access to this entire namespace
    GRANT SELECT ON ALL TABLES IN DATABASE telemetry TO USER `analytics-prod`;
* Option B: Virtual Users (External Authorization) If you use 
authentication_chain
to automatically create virtual users for every login, StarRocks will create a session-based user named 
analytics-prod
. For this "dynamic" approach, you must currently use Apache Ranger to manage the permissions for these virtual users. ────────── Important Considerations 1. Claim Verification: To verify the exact claims your Kubernetes cluster is sending, you can inspect a token from your client pod. Run 
kubectl exec <pod-name> -- cat /var/run/secrets/kubernetes.io/serviceaccount/token
and paste it into jwt.io. Look for the key 
<http://kubernetes.io/serviceaccount/namespace|kubernetes.io/serviceaccount/namespace>
. 2. Security Granularity: By authenticating at the namespace level, you lose the ability to distinguish between different apps (ServiceAccounts) within that same namespace inside StarRocks. If 
App-A
and 
App-B
are both in 
analytics-prod
, they will have identical privileges. 3. Cross-Cluster Networking: Since your StarRocks cluster is in a different K8s cluster, ensure your FEs can reach the 
jwks_url
of the client cluster. If the client cluster is a managed service like EKS or GKE, the OIDC endpoint is usually publicly accessible. If it is on-premise/private, you may need to host a copy of the 
jwks
file locally on the StarRocks FE nodes. Summary of Benefits * Simplified Onboarding: You don't need to create a StarRocks user for every new microservice; you only create one user per Kubernetes namespace. * Namespace-Level Control: You can manage data access at the "team" or "environment" level (e.g., all apps in the 
staging
namespace get access to 
staging_db
). References • administration/user_privs/authentication/jwt_authentication.md • administration/user_privs/authentication/security_integration.md • administration/management/FE_configuration.md References
s
That would be 
<http://kubernetes.io/namespace|kubernetes.io/namespace>
from what I see in a token.
@Rocky If different apps in the same namespace need to map to different users and the app owner for some reason cannot change their service account, what are our options then ?
r
If different applications are forced to share the same Namespace and the same ServiceAccount, they are cryptographically identical from the perspective of a standard Kubernetes token. However, you can still distinguish them in StarRocks by using the Audience (
aud
) field as a unique identifier. Here is how you can set this up: 1. Differentiate at the K8S Pod Level Even if the ServiceAccount is the same, each application can request a token with a unique Audience. In your Kubernetes Deployment YAML for each app, use a 
projected
volume to specify a custom audience: App A Deployment:
Copy code
yaml
spec:
  template:
    spec:
      containers:
      - name: app-a
        volumeMounts:
        - name: sr-token
          mountPath: /var/run/secrets/starrocks
      volumes:
      - name: sr-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              audience: "app-a-identity" # Unique identifier for App A
              expirationSeconds: 3600
App B Deployment:
Copy code
yaml
# ... same as above ...
              audience: "app-b-identity" # Unique identifier for App B
2. Configure StarRocks to use Audience as the Username By default, StarRocks looks at the 
sub
field to determine the username. You can tell it to look at the 
aud
(audience) field instead using the 
principal_field
property. Create a Security Integration that maps the audience directly to the login name:
Copy code
sql
CREATE SECURITY INTEGRATION app_specific_auth
PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "https://<cluster-b-oidc-endpoint>/openid/v1/jwks",
    -- Map the 'aud' claim to the StarRocks username
    "principal_field" = "aud", 
    "required_issuer" = "<https://kubernetes.default.svc.cluster.local>"
);
3. Create Separate Users in StarRocks Now, you can create separate users in StarRocks that match the unique audience strings you defined in Kubernetes:
Copy code
sql
-- Create User for App A
CREATE USER `app-a-identity` IDENTIFIED WITH authentication_jwt;
GRANT SELECT ON db1.* TO USER `app-a-identity`;

-- Create User for App B
CREATE USER `app-b-identity` IDENTIFIED WITH authentication_jwt;
GRANT SELECT, INSERT ON db2.* TO USER `app-b-identity`;
Why this works When App A connects, it sends a JWT where 
aud
is 
app-a-identity
. StarRocks extracts 
app-a-identity
as the username and applies the permissions for that specific user. Even though the 
sub
(ServiceAccount) is the same for both, the 
aud
differentiates them. Alternative: Custom Claims (Advanced) If your Kubernetes environment or OIDC provider allows adding custom claims to the JWT (some managed providers allow this via admission controllers or OIDC proxies), you can set 
principal_field
to any custom key you like (e.g., 
"principal_field" = "custom_app_name"
). Summary of Options If you can change... | Use this 
principal_field
 | Result ServiceAccount | 
sub
(Default) | Unique user per ServiceAccount Namespace | 
<http://kubernetes.io/serviceaccount/namespace|kubernetes.io/serviceaccount/namespace>
| Unique user per Namespace Audience | 
aud
| Unique user per Deployment (Best for your case) References • administration/user_privs/authentication/jwt_authentication.md • administration/user_privs/authentication/security_integration.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/user_privs/authentication/jwt_authentication.md
s
@Rocky Can I combine 
<http://kubernetes.io/namespace|kubernetes.io/namespace>
and the 
aud
audiance field together in the security integration so that it matches on say namespace=xxxx AND audience=yyyy
r
While StarRocks doesn't support a "concatenation" syntax (like 
field1 + field2
) inside the 
principal_field
property, you can achieve the logic of matching on both by using the validation properties in conjunction with the identity property. In StarRocks, a JWT authentication succeeds only if all specified validation checks pass. You can use this to create an "AND" condition: 1. The "Validation AND Identity" Approach If you want to ensure the token belongs to a specific audience AND you want to identify the user by their namespace:
Copy code
sql
CREATE SECURITY INTEGRATION k8s_scoped_auth
PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "...",
    -- This sets the identity (the StarRocks username)
    "principal_field" = "<http://kubernetes.io/serviceaccount/namespace|kubernetes.io/serviceaccount/namespace>", 
    -- This acts as a mandatory filter (The "AND" condition)
    "required_audience" = "my-specific-app-audience",
    "required_issuer" = "<https://kubernetes.default.svc.cluster.local>"
);
How it works: * Condition 1 (Validation): If the JWT does not have 
aud: "my-specific-app-audience"
, the login is rejected immediately. * Condition 2 (Identity): If the audience matches, StarRocks then looks at the 
namespace
field to determine the username (e.g., 
analytics-prod
). 2. Handling Multiple Pairs (The Authentication Chain) If you have multiple apps where 
Namespace-A
must use 
Audience-X
and 
Namespace-B
must use 
Audience-Y
, you can create multiple security integrations and link them in the 
authentication_chain
. 1. Create Integration 1: 
principal_field
= namespace, 
required_audience
= 
Audience-X
2. Create Integration 2: 
principal_field
= namespace, 
required_audience
= 
Audience-Y
3. Enable them:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("authentication_chain" = "int_1, int_2");
When a user attempts to log in, StarRocks will try 
int_1
. If the audience doesn't match 
Audience-X
, it fails 
int_1
and moves to 
int_2
. If the audience matches 
Audience-Y
in 
int_2
, the login succeeds. 3. If you need the Username to be a combination If your goal is for the StarRocks username itself to be 
namespace:audience
, this is currently not supported as a single dynamic property. The 
principal_field
must point to a single existing key in the JWT JSON. Workaround: If you can control the JWT generation (e.g., via a custom OIDC provider or a helper sidecar), you can add a custom claim to the JWT called 
sr_identity
that contains the pre-concatenated string 
"namespace:audience"
and then set 
"principal_field" = "sr_identity"
. Summary * To filter by both: Use 
principal_field
for the name and 
required_audience
for the constraint. * To distinguish different apps in one namespace: Use 
aud
as the 
principal_field
(as discussed in the previous turn). * To strictly "AND" multiple values: Use the 
authentication_chain
with multiple specific integrations. References • administration/user_privs/authentication/security_integration.md • administration/user_privs/authentication/jwt_authentication.md • administration/management/FE_configuration.md References • administration/user_privs/authentication/jwt_authentication.md • administration/user_privs/authentication/security_integration.md • administration/management/FE_configuration.md
s
@Rocky If I have N security integration for the N different K8S cluster client apps can run on, does it mean StarRocks can potentially test an incoming JWT token against all N integration until one of them succeed ?
r
Yes, StarRocks will test the incoming JWT against each security integration defined in your 
authentication_chain
sequentially until it finds a match or exhausts the list. Here is exactly how that process works and what you should consider: 1. Sequential Evaluation When a client attempts to connect using a JWT, StarRocks follows the order defined in the 
authentication_chain
FE configuration:
Copy code
sql
-- Example: StarRocks checks int_k8s_cluster_1 first, then int_k8s_cluster_2
ADMIN SET FRONTEND CONFIG ("authentication_chain" = "int_k8s_cluster_1, int_k8s_cluster_2, ... , int_k8s_cluster_N");
For each integration in the list: 1. Validation: It checks if the JWT's 
iss
(issuer) and 
aud
(audience) match the 
required_issuer
and 
required_audience
defined in that specific integration. 2. Verification: If the fields match, it attempts to verify the cryptographic signature using the 
jwks_url
associated with that integration. 3. Result: * If successful, the user is logged in. * If it fails (wrong issuer, expired token, or signature mismatch), StarRocks silently moves to the next integration in the chain. 2. Performance Implications While StarRocks can handle multiple integrations, there are two performance factors to keep in mind: * Signature Verification: Checking a signature is a CPU-intensive task. If you have 50 integrations and the correct one is the 50th, StarRocks might perform multiple validation checks before succeeding. However, it usually fails quickly on the 
issuer
check before doing the heavy lifting of signature verification. * JWKS Fetching: StarRocks caches the public keys from 
jwks_url
. If you have $N$ integrations, the FE must maintain $N$ sets of public keys in memory and periodically refresh them. 3. Best Practices for "N" Clusters If you are managing a large number of Kubernetes clusters ($N$ is large), consider these optimizations: * Order by Frequency: Place the most active clusters at the beginning of the 
authentication_chain
to minimize the number of checks for the majority of login attempts. * Unique Issuers: Ensure every Kubernetes cluster has a unique 
iss
(issuer) claim. This allows StarRocks to instantly skip integrations that don't match the issuer string without performing expensive cryptographic operations. * Centralized OIDC (Recommended): If $N$ becomes too large to manage, it is often better to use an intermediate OIDC provider (like Okta, Keycloak, or Authelia). All K8s clusters authenticate to the OIDC provider, and StarRocks only needs one security integration pointing to that single central provider. Important Note on Authorization When using multiple security integrations via the 
authentication_chain
, StarRocks typically creates virtual users (session-based). In this mode: 1. Native 
GRANT
statements (e.g., 
GRANT SELECT ON...
) might not work as expected because the user doesn't "exist" until they log in. 2. Apache Ranger is the recommended way to manage permissions for these dynamic users across multiple clusters/integrations. References • administration/user_privs/authentication/jwt_authentication.md • administration/user_privs/authentication/security_integration.md • administration/management/FE_configuration.md References • administration/user_privs/authentication/security_integration.md • administration/management/FE_configuration.md • best_practices/authentication_authorization.md
s
@Rocky For HTTP based request, app just need to 
Authorization: Bearer ${k8s_token}
right ?
r
Actually, it is not just 
Authorization: Bearer <token>
. While the industry standard for JWT is the 
Bearer
scheme, StarRocks follows a specific standard for its RESTful APIs and SQL over HTTP. For the HTTP SQL API If you are using the HTTP SQL API (e.g., 
POST /api/v1/catalogs/default_catalog/sql
), the current documentation and implementation primarily emphasize Basic Authentication (
Authorization: Basic <base64(user:pass)>
). However, for JWT-based authentication via HTTP (starting from StarRocks v3.5.0), the behavior depends on the endpoint: 1. Direct JWT Support: When using JWT authentication for HTTP-based services (like the Web UI or JDBC with OAuth), StarRocks expects the token. 2. The Header Format: In StarRocks, when passing a JWT token for authentication in HTTP headers (similar to how the MySQL client uses it), you typically use:
Copy code
http
    Authorization: Bearer <your_k8s_jwt_token>
BUT, there is a crucial catch: The StarRocks FE must be configured to recognize the user and the integration. Since JWT users in StarRocks often map to a specific "principal" (like the 
aud
or 
sub
we discussed), you don't necessarily provide a password. Important: Username in the Request When you use a JWT, StarRocks still needs to know which user you are claiming to be so it can look up the correct Security Integration rules. Even with a Bearer token, if you are using the HTTP SQL API, you often still need to identify the user. If your Security Integration uses 
principal_field = "aud"
, and your token has 
aud: "app-a"
, StarRocks will attempt to match the incoming request to the StarRocks user named 
app-a
. Summary for your K8S Apps For your K8S apps to talk to StarRocks over HTTP using their service account tokens: 1. Version: Ensure you are on v3.5.0 or later. 2. Headers:
Copy code
bash
    curl -X POST "http://<fe_ip>:8030/api/v1/catalogs/default_catalog/sql" \
         -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
         -H "Content-Type: application/json" \
         -d '{"query": "SELECT * FROM my_table"}'
3. User Mapping: Ensure the 
principal_field
in your 
SECURITY INTEGRATION
correctly extracts the string that matches a 
CREATE USER '...' IDENTIFIED WITH authentication_jwt
user in StarRocks. Note: If you find the HTTP SQL API still strictly requires Basic Auth in your specific minor version, the standard workaround is to use the JDBC driver which has explicit support for OAuth2/JWT flows and can be used by most application runtimes like Java, Python (via SQLAlchemy), or Go. References • administration/user_privs/authentication/jwt_authentication.md • sql-reference/http_sql_api.md • administration/user_privs/authentication/security_integration.md References • administration/user_privs/authentication/jwt_authentication.md • developers/code-style-guides/restful-api-standard.md • administration/user_privs/authentication/oauth2_authentication.md • sql-reference/http_sql_api.md • administration/management/FE_configuration.md • administration/user_privs/authentication/security_integration.md
s
@Rocky I'm on version 4.0.x do all HTTP based endpoint support JWT ?
r
In StarRocks v4.0.x, while JWT support has been significantly expanded, it is important to distinguish between official support and endpoint compatibility. The short answer is: Not all HTTP-based endpoints support JWT "Bearer" tokens yet. Most core user-facing query interfaces do, but legacy or specialized system APIs might still require Basic Auth. Here is the breakdown of JWT/OAuth2 support for HTTP endpoints in 4.0.x: 1. Supported: Query & User Interfaces For these endpoints, you can use JWT/OAuth2 authentication (typically via a 
Bearer
token or the OIDC flow): * Web UI: Fully supports OAuth2/OIDC login. * JDBC over HTTP: The JDBC driver supports OAuth2 and JWT flows when connecting via the HTTP protocol. * Iceberg REST Catalog (New in 4.0): Supports "JWT Identity Passthrough," where StarRocks receives a JWT from the client and forwards it to the Iceberg REST Catalog to ensure end-to-end identity. * HTTP SQL API (
/api/v1/.../sql
): While the documentation still emphasizes 
Basic
Auth, version 3.5+ and 4.0.x have unified much of the authentication logic. If you have a 
SECURITY INTEGRATION
configured and a user identified with 
authentication_jwt
, the system can often process the token. However, Basic Auth remains the most stable method for this specific endpoint in many automated scripts. 2. Partial/Legacy: Data Loading & System APIs * Stream Load (
/api/{db}/{table}/_stream_load
): Historically, Stream Load has relied strictly on 
Basic
authentication. While there are architectural moves toward unified auth, it is best to continue using 
Basic
(or a Secret Key if using a Cloud-native load) for Stream Load to ensure compatibility across all 4.0.x patches. * Metrics/Health (
/metrics
, 
/api/health
): These are typically unauthenticated or rely on a simple whitelist/password and do not usually process JWTs. How to check for your specific 4.0.x version Since v4.0 is a major release with frequent minor updates, you can verify if your specific endpoint supports JWT by checking the 
authentication_chain
. 1. Define the Integration:
Copy code
sql
    CREATE SECURITY INTEGRATION k8s_auth PROPERTIES ("type" = "authentication_jwt", ...);
2. Enable the Chain:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("authentication_chain" = "k8s_auth");
3. Test the Endpoint: Try a 
curl
request with your K8S token:
Copy code
bash
    curl -H "Authorization: Bearer $TOKEN" http://<FE_IP>:8030/api/v1/catalogs/default_catalog/sql -d '{"query": "SELECT 1"}'
If it returns a 
401 Unauthorized
, that specific endpoint still requires Basic Auth in your current sub-version. Summary of 4.0.x Improvements StarRocks 4.0.x is designed to be "Catalog-Centric," meaning the focus of JWT support is on flowing your identity from the client -> StarRocks -> External Data Lakes (like Iceberg). This is the most robust implementation of JWT in the system to date. Recommendation: For programmatic HTTP access (like the SQL API or Stream Load) where JWT isn't working, the standard practice is to use the Service Account's credentials via Basic Auth, or use the JDBC driver which handles the JWT handshake more reliably than the raw HTTP endpoints do. References • administration/user_privs/authentication/jwt_authentication.md • administration/user_privs/authentication/security_integration.md • sql-reference/http_sql_api.md • administration/user_privs/authentication/oauth2_authentication.md
s
@Rocky Are each incoming request rechecked everytime, is there a token check cache to avoid hitting the OIDC provider on every request?
r
No, StarRocks does not hit the OIDC provider on every request. It uses a highly efficient local validation process. Here is how the caching and validation work to ensure performance isn't degraded when you have multiple Kubernetes clusters/integrations: 1. The "Public Key" Cache (JWKS) StarRocks does not send the token to your OIDC provider for validation (which would be an "introspection" flow). Instead, it performs Local Signature Verification. * How it works: When you define a 
jwks_url
, StarRocks (the FE) fetches the public keys from that URL and caches them in memory. * Refresh Behavior: StarRocks periodically refreshes this cache in the background. If your K8S cluster rotates its signing keys, StarRocks will eventually pick up the new keys without manual intervention. * Result: Every incoming request is validated using the keys already held in the FE's memory. No network call is made to the OIDC provider during the query path. 2. Token Check "Cache" vs. Re-validation While there isn't a "cache of successful tokens" (e.g., Token A was valid 5 seconds ago, so skip check), the re-validation is extremely fast because it is purely a CPU-bound cryptographic check: * Stateless (HTTP): For the HTTP SQL API, every request carries the Bearer token. StarRocks performs a local signature check + expiration check. This typically takes only a few microseconds. * Stateful (MySQL Protocol): If you are using a MySQL client with JWT, the authentication happens only once during the handshake. After the session is established, StarRocks does not re-check the token for every query in that same session. 3. Impact of Having "N" Integrations If you have 10 different integrations in your 
authentication_chain
, StarRocks will iterate through them. However, it is optimized to fail fast: 1. Issuer Check: It first looks at the 
iss
claim in the JWT. If the token says 
issuer: cluster-A
and the integration is for 
cluster-B
, StarRocks skips it immediately without even attempting the heavy cryptographic signature check. 2. Key ID (
kid
) Match: It looks for a matching 
kid
in its cached keys. If the token’s key ID doesn't exist in that integration's cache, it moves on. Performance Summary Action | Hits OIDC Provider? | Latency Cost Initial Key Fetch | Yes (Once) | High (Network) Key Refresh | Yes (Background) | Zero (Non-blocking) Token Validation | No | Very Low (CPU) Issuer Mismatch | No | Negligible Best Practices for Your Setup * Unique Issuers: Ensure each of your N Kubernetes clusters has a unique 
issuer
string. This allows StarRocks to instantly identify which integration to use. * Local JWKS File: If you are worried about the FE's ability to reach the OIDC provider over the network, you can download the public key to a local file and point 
jwks_url
to a local path (e.g., 
file:///path/to/keys.json
). This removes the OIDC provider dependency entirely during runtime. References • administration/user_privs/authentication/jwt_authentication.md • administration/user_privs/authentication/security_integration.md • administration/management/FE_configuration.md References • administration/user_privs/authentication/jwt_authentication.md • administration/management/FE_configuration.md • best_practices/authentication_authorization.md • using_starrocks/caching/query_cache.md
s
@Rocky Do HTTP Stream Load and HTTP SQL Query support these JWT authentiction?
r
In StarRocks v4.0.x, both the HTTP SQL Query API and HTTP Stream Load support JWT authentication, but the implementation details differ slightly based on the endpoint and how you have configured your cluster. 1. HTTP SQL API (
/api/v1/.../sql
) This endpoint fully supports JWT authentication via the standard Bearer token header. * Header: 
Authorization: Bearer <your_jwt_token>
* How it works: When you send a request to the FE, the StarRocks FE checks the 
authentication_chain
. If you have a 
SECURITY INTEGRATION
of type 
authentication_jwt
or a user 
IDENTIFIED WITH authentication_jwt
, the FE validates the token's signature (using the cached JWKS) and maps the 
principal_field
(usually 
sub
or 
aud
) to a StarRocks user. * Advantage: This allows you to avoid hardcoding database passwords in your application code or Kubernetes manifests. 2. HTTP Stream Load (
/api/{db}/{table}/_stream_load
) Stream Load also supports JWT authentication in v4.0.x, but with a specific architectural flow: * Header: 
Authorization: Bearer <your_jwt_token>
* The Flow: 1. Your application sends the 
PUT
request with the Bearer token to the FE. 2. The FE validates the JWT and determines the identity of the user. 3. The FE then redirects the client to a BE (Backend) node to upload the actual data. * Important Note: To ensure this works seamlessly across redirects, you should use a client that can handle the 
307 Temporary Redirect
while retaining the Authorization header. Some tools (like certain versions of 
curl
or older HTTP libraries) might drop the 
Authorization
header during a redirect for security reasons. In 
curl
, you would use the 
--location-trusted
flag:
Copy code
bash
    curl --location-trusted -X PUT \
         -H "Authorization: Bearer $K8S_TOKEN" \
         -T data.csv \
         http://<fe_ip>:8030/api/mydb/mytable/_stream_load
3. Requirements for v4.0.x For either endpoint to accept your K8S token, you must have the following set up: 1. Security Integration:
Copy code
sql
    CREATE SECURITY INTEGRATION k8s_integration
    PROPERTIES (
        "type" = "authentication_jwt",
        "jwks_url" = "<https://kubernetes.default.svc/openid/v1/jwks>", -- or your IdP URL
        "jwt_principal_field" = "sub" -- Mapping the 'sub' claim to the SR username
    );
2. User Mapping: The user specified in the 
sub
claim of your token must exist in StarRocks:
Copy code
sql
    CREATE USER `system:serviceaccount:namespace:app-name`
    IDENTIFIED WITH authentication_jwt;
3. Authentication Chain: Ensure the FE is configured to use the integration:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("authentication_chain" = "k8s_integration,native");
Summary Comparison Endpoint | v4.0.x JWT Support | Recommended Header HTTP SQL API | Official | 
Authorization: Bearer <token>
Stream Load | Supported | 
Authorization: Bearer <token>
MySQL Protocol | Official | Handled via OIDC plugin/JDBC Rocky's Tip: While Bearer tokens are supported, if you find that a specific client library struggles with redirects or header persistence during Stream Load, you can also use OAuth2 Client Credentials to obtain a temporary session, though the direct Bearer token approach is the most "Kubernetes-native" way to go. References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/http_sql_api.md • loading/StreamLoad.md
s
@Rocky Previously you mentionned I could create user based on the 
sub
field like this:
Copy code
-- The username must match the 'principal_field' (usually the 'sub' claim)
    CREATE USER `system:serviceaccount:default:my-app-sa`
    IDENTIFIED WITH k8s_auth; -- 'k8s_auth' is the name of your Security Integration
What happens if I need to identify this default:my-app-sa user from more than one security integration ? The same client app will run from many different k8s cluster but they all need to map to the same user because each will expose identical 
sub
field. Can you identify using multiple integration?
r
Yes, you can identify a user from multiple security integrations, but there is a nuance in how StarRocks handles the mapping and permissions. In StarRocks v4.0.x, if you have multiple Kubernetes clusters with different issuers/JWKS URLs, you cannot achieve this by simply creating a single native user (the 
CREATE USER
method), because a native user's JWT properties only support one 
jwks_url
. Instead, the recommended approach for your "multiple clusters, same identity" scenario is to use Security Integrations combined with Role-to-Group mapping. The Solution: Multi-Integration Chain To allow the same Service Account from different clusters to access StarRocks with the same permissions, follow this architecture: 1. Define a Security Integration for each cluster Since each K8s cluster has its own public key set (JWKS), you must define them separately:
Copy code
sql
-- For Cluster A
CREATE SECURITY INTEGRATION k8s_cluster_a PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "<https://cluster-a.example.com/openid/v1/jwks>",
    "principal_field" = "sub"
);

-- For Cluster B
CREATE SECURITY INTEGRATION k8s_cluster_b PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "<https://cluster-b.example.com/openid/v1/jwks>",
    "principal_field" = "sub"
);
2. Configure the Authentication Chain Set the FE to check all integrations in order. When a token arrives, StarRocks will try them one by one. If it matches the issuer and signature of Cluster B, it will stop there.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("authentication_chain" = "k8s_cluster_a, k8s_cluster_b, native");
3. Handling Permissions (The "Same User" Part) When using the 
SECURITY INTEGRATION
method, StarRocks creates a session-scoped virtual user (dummy user) rather than looking up a native user record. To give this "dummy user" permissions natively without pre-creating it 10 times, you use Group Mapping: 1. Grant to a Role: Create a role that has all the permissions your app needs.
Copy code
sql
    CREATE ROLE app_runtime_role;
    GRANT SELECT ON ALL TABLES IN DATABASE mydb TO ROLE app_runtime_role;
2. Map the Identity to the Role: You can map the specific 
sub
(your Service Account name) to this role. In v4.0.x, you can use the Group Provider feature or the security integration's ability to map external groups to internal roles. * If your K8s JWT includes a 
groups
claim, you can map that group to the 
app_runtime_role
. * If you only have the 
sub
, you can use StarRocks' native RBAC to grant the role to the user name that will be generated. StarRocks allows you to grant roles to a user identity even if the user hasn't logged in yet:
Copy code
sql
    -- Even though the user is "virtual", the name is consistent.
    -- Grant the role to the name that the principal_field will produce.
    GRANT app_runtime_role TO USER `system:serviceaccount:default:my-app-sa`;
Why this works * Authentication: The 
authentication_chain
handles the fact that keys and issuers are different across clusters. * Identity: Both integrations produce the exact same string for the user identity (e.g., 
system:serviceaccount:default:my-app-sa
) because they use the same 
principal_field
mapping. * Authorization: Because the session user name is identical regardless of which cluster it came from, StarRocks applies the permissions you granted to that specific user string. Summary of Rules * One Native User = One Integration: A user created via 
CREATE USER ... IDENTIFIED WITH authentication_jwt
is hard-linked to one set of JWT properties. * Multiple Integrations = Virtual Users: Using 
CREATE SECURITY INTEGRATION
allows multiple clusters to "hand off" an identity to StarRocks. * Consistent Naming: As long as your K8s Service Account names are the same, they will resolve to the same StarRocks identity and inherit the same roles. References
s
@Rocky The GRANT fails, why is that ? 
aaa-bbb
is a k8s namespace with a dash. We cannot change this.
Copy code
StarRocks> GRANT `k8s_sa_ucc_core_default` TO USER `system:serviceaccount:aaa-bbb:default`;
ERROR 1064 (HY000): Getting analyzing error. Detail message: invalid user name: system:serviceaccount:aaa-bbb:default.
r
The error you are encountering occurs because StarRocks usernames have strict naming conventions that, by default, do not allow colons (
:
) or dashes (
-
). According to StarRocks system limits, usernames can only consist of letters (a-z, A-Z), digits (0-9), and underscores (
_
). Since Kubernetes Service Account subjects (
sub
) are automatically formatted as 
system:serviceaccount:namespace:serviceaccount-name
, they almost always violate this rule if the namespace or service account contains a dash. To fix this and successfully grant permissions, you have two primary options: Option 1: Use Single Quotes and Host Suffix (Recommended) The StarRocks parser is often stricter with backticks (which treat the string as a SQL identifier) than with single quotes (which treat it as a string literal). To reference a user identity that contains special characters, you must include the host part (usually 
'%'
) and use single quotes for both the username and the host. Try this exact syntax:
Copy code
sql
GRANT `k8s_sa_ucc_core_default` TO USER 'system:serviceaccount:aaa-bbb:default'@'%';
*Note: Ensure you use single quotes (
'
) for the user identity, not backticks.* Option 2: Grant to an "EXTERNAL GROUP" instead If you are using a Security Integration (the virtual user approach), the recommended way to handle Kubernetes identities is to grant roles to an External Group rather than the individual user. StarRocks' 
EXTERNAL GROUP
identifier is more flexible with special characters. Kubernetes JWTs typically include a 
groups
claim. You can grant your role to the group representing the namespace:
Copy code
sql
-- Grant to the entire namespace group (which also has dashes/colons)
GRANT `k8s_sa_ucc_core_default` TO EXTERNAL GROUP 'system:serviceaccounts:aaa-bbb';
By granting to the external group, any service account within that namespace that logs in via JWT will automatically inherit the role. Option 3: Map to a "Clean" Native User If you prefer to use the 
CREATE USER
(Native User) method, you cannot use the full 
sub
as the username because of the naming restriction. Instead: 1. Use a custom claim: If your OIDC provider/Kubernetes allows it, add a custom claim to the JWT (e.g., 
"starrocks_user": "ucc_core_sa"
) that contains only letters and underscores. 2. Point the Security Integration to that claim:
Copy code
sql
    CREATE SECURITY INTEGRATION k8s_integration PROPERTIES (
        ...
        "jwt_principal_field" = "starrocks_user"
    );
3. Grant to the clean name:
Copy code
sql
    GRANT `k8s_sa_ucc_core_default` TO USER `ucc_core_sa`;
Summary of the Naming Rule Character | Allowed in Native Username? | Allowed in GRANT (Quoted)? 
a-z, A-Z, 0-9, _
| Yes | Yes 
-
(Dash) | No | Yes (with 
'user'@'host'
) 
:
(Colon) | No | Yes (with 
'user'@'host'
) Rocky's Recommendation: Use Option 1 first. If the 
GRANT
still fails with an "invalid user name" error even with single quotes and the 
@'%'
suffix, it means your StarRocks version is strictly enforcing the identifier check on the username component. In that case, Option 2 (External Groups) is the standard architectural workaround for Kubernetes integrations. References • sql-reference/System_limit.md • sql-reference/sql-statements/account-management/GRANT.md • administration/user_privs/authentication/security_integration.md • administration/user_privs/authorization/user_privs.md References • sql-reference/System_limit.md
s
Anyone managed to get JWT authentication working from kubernetes service token ? The 
sub
field format of systemserviceaccountk8s-namespace:k8s-service-account-name is not compatible with StarRocks USER name format. Adding custom claims to these tokens is not possible in k8s. Can it it work or we have to fallback shared credentials using native authentication ?
@Rocky For this type of CREATE USER command, instead of repeating all these authentication_jwt parameters, can you instead reference a pre-existing SECURITY INTEGRATION ?
Copy code
CREATE USER tom IDENTIFIED WITH authentication_jwt AS
'{
  "jwks_url": "<http://localhost:38080/realms/master/protocol/jwt/certs>",
  "principal_field": "preferred_username",
  "required_issuer": "<http://localhost:38080/realms/master>",
  "required_audience": "starrocks"
}';
r
Yes, you can reference a pre-existing Security Integration to avoid repeating the long JSON property block every time you create a user. In StarRocks, once a Security Integration is created, its name effectively becomes a new "Authentication Method" that you can use in the 
CREATE USER
statement. There are two ways to achieve your goal, depending on whether you want to manually manage the user or let StarRocks handle it automatically. Option 1: Manually Create the User (Referencing the Integration) If you have already created a Security Integration (e.g., named 
k8s_auth
), you can link a native user to it using this simplified syntax:
Copy code
sql
-- 1. Create the integration once
CREATE SECURITY INTEGRATION k8s_auth
PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "<http://localhost:38080/realms/master/protocol/jwt/certs>",
    "principal_field" = "preferred_username",
    "required_issuer" = "<http://localhost:38080/realms/master>",
    "required_audience" = "starrocks"
);

-- 2. Create the user by referencing the integration name
-- This inherits all the parameters from 'k8s_auth'
CREATE USER tom IDENTIFIED WITH k8s_auth;
Option 2: Automatic "Virtual Users" (Recommended) The primary purpose of a 
SECURITY INTEGRATION
is to remove the need for 
CREATE USER
entirely. If you configure your cluster's 
authentication_chain
, StarRocks will automatically authenticate any user who presents a valid JWT from that provider and create a "virtual user" for that session. 1. Configure the Chain:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("authentication_chain" = "k8s_auth, native");
2. Grant Permissions: Instead of creating the user 
tom
, you simply grant permissions to the identity that will be produced by the 
principal_field
(in your case, the 
preferred_username
claim):
Copy code
sql
    GRANT ALL PRIVILEGES ON DATABASE mydb TO USER tom;
When "tom" logs in with a valid JWT, StarRocks sees the integration in the chain, validates the token, and maps the session to the "tom" identity with all his granted privileges. Option 3: Using FE Configuration Defaults If you only have one JWT provider for the entire cluster and you still prefer the 
CREATE USER
approach, you can set global defaults in your 
fe.conf
file:
Copy code
properties
# fe.conf
jwt_jwks_url = <http://localhost:38080/realms/master/protocol/jwt/certs>
jwt_principal_field = preferred_username
After restarting the FE, you can simply run:
Copy code
sql
CREATE USER tom IDENTIFIED WITH authentication_jwt;
This will automatically use the parameters defined in your configuration file. Summary * To avoid code repetition: Use Option 1 (
IDENTIFIED WITH <integration_name>
). * For the best architecture: Use Option 2 (Security Integration + 
authentication_chain
) so you don't have to manage 
CREATE USER
and 
DROP USER
cycles at all. Rocky's Tip: Using a named Security Integration is much cleaner for maintenance. If your JWKS URL or Audience changes, you only need to 
ALTER SECURITY INTEGRATION
in one place, and all users linked to it (or virtual users logging in through it) will immediately use the new settings. References • sql-reference/sql-statements/account-management/CREATE_USER.md • administration/user_privs/authentication/security_integration.md
s
@Rocky My authentication chain would produce username with invalid character like 
:
so i cannot assign permission to these:
Copy code
StarRocks> GRANT ALL PRIVILEGES ON DATABASE mydb TO USER 'system:serviceaccount:namespace:sa';
ERROR 1064 (HY000): Getting analyzing error. Detail message: invalid user name: system:serviceaccount:namespace:sa.
r
The error occurs because StarRocks enforces strict naming conventions for "Native Users" (users whose identities are stored in the internal metadata). By default, a username can only contain letters, digits, and underscores. Characters like colons (
:
) and dashes (
-
)—which are standard in Kubernetes 
sub
claims—are rejected by the SQL analyzer when used in a 
GRANT ... TO USER
statement. To solve this for your multi-integration K8s setup, you should use Role-to-Group mapping or the External Group feature. This bypasses the naming restriction of native users. The Solution: Use 
EXTERNAL GROUP
 Instead of granting privileges to a specific 
USER
(which triggers the naming check), you can grant the role to an External Group. StarRocks allows more flexibility in the naming of external groups because they are treated as string literals from your IDP. 1. Identify the Group in your K8s JWT Kubernetes Service Account tokens usually include a 
groups
claim. For example: • 
system:serviceaccounts
• 
system:serviceaccounts:<namespace>
2. Grant to the External Group Instead of the specific service account, grant permissions to the group representing the namespace. StarRocks' 
GRANT ... TO EXTERNAL GROUP
syntax is designed for this:
Copy code
sql
-- This string is treated as a literal and can contain colons/dashes
GRANT `k8s_sa_ucc_core_default` TO EXTERNAL GROUP 'system:serviceaccounts:aaa-bbb';
When any service account from the 
aaa-bbb
namespace logs in, StarRocks sees the group in the JWT, matches it to this grant, and the user automatically inherits the 
k8s_sa_ucc_core_default
role. ────────── Alternative: Mapping 
sub
to a "Clean" Name If you absolutely must grant to the individual Service Account and cannot use groups, you have two workarounds: Option A: Use a Custom Principal Field (Recommended) If you can configure your K8s OIDC issuer (or your identity proxy) to include a "clean" alias claim in the JWT (e.g., 
starrocks_name: "my_app_sa"
), you can point your Security Integration to that field:
Copy code
sql
CREATE SECURITY INTEGRATION k8s_auth PROPERTIES (
    "type" = "authentication_jwt",
    "principal_field" = "starrocks_name", -- Use the clean field instead of 'sub'
    ...
);

-- Now you can grant to the clean name
GRANT `my_role` TO USER `my_app_sa`;
Option B: The "Session Identity" Trick If you are on StarRocks v3.3.3 or later, the 
GRANT
statement is sometimes more lenient if you use the full 
user_identity
format with single quotes and the host suffix. Try this exact syntax:
Copy code
sql
GRANT `k8s_sa_ucc_core_default` TO USER 'system:serviceaccount:aaa-bbb:default'@'%';
Note: You must include the 
'@'%'
part. Using single quotes tells the parser to treat the username as a string literal rather than a SQL identifier. Why this is the "Rocky" Way 1. Consistency: By using 
EXTERNAL GROUP
, you don't need to 
CREATE USER
for every single app instance. 2. Scalability: If you deploy the same app to 50 namespaces, you can grant to the group 
system:serviceaccounts
once, and all of them are covered. 3. No Naming Conflicts: You avoid the headache of trying to "clean" colons and dashes out of K8s subjects. Summary of the Fix: If 
GRANT ... TO USER
fails with 
invalid user name
, switch to: 
GRANT <role> TO EXTERNAL GROUP '<full-sub-or-group-string>';
References • sql-reference/System_limit.md • administration/user_privs/authorization/User_privilege.md • administration/user_privs/authentication/security_integration.md • sql-reference/sql-statements/account-management/GRANT.md References
s
@Rocky Show me a full working example using EXTERNAL GROUP for JWT auth
r
To handle Kubernetes service accounts or any identity containing special characters like colons (
:
) or dashes (
-
), the most robust approach is to use Security Integrations paired with External Groups. This method avoids the "invalid username" error because you grant permissions to a group string (which is a literal) rather than a native username identifier (which is subject to strict SQL naming rules). The Full Example This example assumes you have a Kubernetes cluster providing JWTs where the 
groups
claim contains strings like 
system:serviceaccounts:aaa-bbb
. 1. Create the Security Integration The Security Integration defines how StarRocks validates the JWT and which claim contains the group information.
Copy code
sql
CREATE SECURITY INTEGRATION k8s_jwt_integration
PROPERTIES (
    "type" = "authentication_jwt",
    "jwks_url" = "<https://kubernetes.default.svc/openid/v1/jwks>",
    "principal_field" = "sub",           -- The unique ID (e.g., system:serviceaccount:...)
    "groups_field" = "groups",           -- THE KEY: identifies the claim containing group list
    "required_issuer" = "<https://kubernetes.default.svc>",
    "required_audience" = "starrocks"
);
2. Configure the Authentication Chain Tell StarRocks to use this integration when a user attempts to log in.
Copy code
sql
-- Enable the integration in the auth chain
ADMIN SET FRONTEND CONFIG ("authentication_chain" = "k8s_jwt_integration, native");
3. Create a Role and Grant to the External Group Instead of granting to the user 
system:serviceaccount:aaa-bbb:default
, you grant to the group that represents the namespace. The string literal in single quotes allows colons and dashes.
Copy code
sql
-- 1. Create a functional role
CREATE ROLE k8s_app_role;

-- 2. Grant permissions to the role
GRANT SELECT ON ALL TABLES IN DATABASE mydb TO ROLE k8s_app_role;

-- 3. Map the role to the K8s namespace group
-- This string must match the 'groups' claim in your K8s JWT
GRANT k8s_app_role TO EXTERNAL GROUP 'system:serviceaccounts:aaa-bbb';
4. How the User Logs In The application in the 
aaa-bbb
namespace logs in using its Service Account token. Because it belongs to the 
system:serviceaccounts:aaa-bbb
group (in the JWT), it automatically inherits 
k8s_app_role
. Bash Example:
Copy code
bash
# The username passed here just needs to match the 'sub' field in the token
# StarRocks treats it as a 'virtual user' and doesn't check the naming convention
mysql -h <fe_ip> -P 9030 \
  --authentication-openid-connect-client-id-token-file=/var/run/secrets/kubernetes.io/serviceaccount/token \
  -u "system:serviceaccount:aaa-bbb:default"
Why this works: 1. Virtual Users: When using a Security Integration, StarRocks creates a "virtual user" for the session. These users are not checked against the strict 
[a-zA-Z0-9_]
native username rule during login. 2. String Literals: The 
EXTERNAL GROUP
syntax takes a string literal 
'...'
. This allows you to include any character (colons, dashes, etc.) that your Identity Provider (Kubernetes) produces. 3. Scalability: You don't need to run 
GRANT
for every single Service Account. You grant it once to the namespace group, and every service account in that namespace is automatically authorized. Troubleshooting the "Invalid User Name" Error If you still get the error when running the 
GRANT
statement, ensure: • You are using 
GRANT ... TO EXTERNAL GROUP '...'
(with single quotes). • You are not using 
GRANT ... TO USER '...'
for these specific identities, as StarRocks tries to validate 
USER
identities against native naming rules. Rocky's Tip: If your StarRocks version is earlier than v3.3, the 
groups_field
property might not be available. In that case, you should upgrade to v3.5+ which has full support for JWT-based RBAC. References • administration/user_privs/authentication/security_integration.md
Open in Slack
PreviousNext
Powered by