http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> how could we integrate keycloak rba...
# questions-and-troubleshooting
k
@Rocky how could we integrate keycloak rbac with starrocks authn and authz
r
Integrating Keycloak Role-Based Access Control (RBAC) with StarRocks requires a two-part setup: configuring Keycloak as an OpenID Connect (OIDC) provider for authentication and leveraging Group Providers or Apache Ranger for authorization mapping. Starting from v3.5.0, StarRocks supports native OIDC and centralized Group Providers. 1. Authentication (Authn): Keycloak OIDC To enable Keycloak login, you create a 
SECURITY INTEGRATION
in StarRocks. This allows users to authenticate via the Keycloak browser flow (for Web UI/JDBC) or JWT tokens. Steps: 1. In Keycloak, create a Client for StarRocks and obtain the 
Client ID
and 
Secret
. 2. In StarRocks, create the security integration:
Copy code
sql
   CREATE SECURITY INTEGRATION keycloak_oidc
   PROPERTIES (
       "type" = "authentication_oauth2",
       "auth_server_url" = "https://<keycloak_host>/realms/<realm>/protocol/openid-connect/auth",
       "token_server_url" = "https://<keycloak_host>/realms/<realm>/protocol/openid-connect/token",
       "jwks_url" = "https://<keycloak_host>/realms/<realm>/protocol/openid-connect/certs",
       "client_id" = "<client_id>",
       "client_secret" = "<client_secret>",
       "redirect_url" = "http://<fe_host>:<fe_http_port>/api/oauth2",
       "principal_field" = "preferred_username" -- Maps Keycloak user to StarRocks user
   );
3. Set the authentication chain in 
fe.conf
(or via SQL) to include this integration:
Copy code
sql
   ADMIN SET FRONTEND CONFIG ("authentication_chain" = "keycloak_oidc, native");
2. Authorization (Authz): Mapping Keycloak Roles While StarRocks can authenticate the user via OIDC, it does not currently have a "JWT Claim" Group Provider to automatically extract roles from a Keycloak token claim. You have three main paths to achieve RBAC integration: Option A: Apache Ranger (Recommended for Enterprise) This is the most robust way to synchronize Keycloak RBAC with StarRocks. * Flow: Keycloak (or its LDAP/AD source) syncs users and groups to Apache Ranger. * Integration: StarRocks is configured to use the Ranger plugin for all authorization checks (
access_control = ranger
in 
fe.conf
). * Benefit: When a user logs in via Keycloak, StarRocks asks Ranger if that user/group has permission for the query. Option B: LDAP Group Provider (Bridge) If your Keycloak is connected to an LDAP/Active Directory backend, or if you use Keycloak's LDAP storage provider: * Flow: Use Keycloak for OIDC Authn, but configure an LDAP Group Provider in StarRocks for Authz. * SQL Example:
Copy code
sql
   CREATE GROUP PROVIDER keycloak_groups
   PROPERTIES (
       "type" = "ldap",
       "ldap_conn_url" = "ldap://<keycloak_ldap_host>:389",
       ... -- LDAP search configs to find user groups
   );

   -- Link the groups to the OIDC login
   ALTER SECURITY INTEGRATION keycloak_oidc SET ("group_provider" = "keycloak_groups");
* Benefit: StarRocks will automatically fetch the user's groups from LDAP upon OIDC login and map them to StarRocks Roles. Option C: Manual/File-based Mapping For smaller environments where groups change infrequently: * Flow: Use the 
FILE
group provider. * Setup: Create a file in 
fe/conf
defining 
group_name:user1,user2
. * Integration: Map these groups to StarRocks roles using 
GRANT <role> TO GROUP <group_name>
. Summary of Requirements * StarRocks Version: v3.5.0 or later is required for the OIDC security integration and Group Provider features. * Protocol: Use the MySQL 9.2+ client or compatible JDBC drivers to support the OIDC authentication plugin. * Limitations: Native StarRocks RBAC currently cannot read "groups" claims directly from a JWT token without a Group Provider (LDAP/File) or Apache Ranger acting as the bridge. References • administration/management/FE_configuration.md
k
@Rocky does starrocks support http based requests vs direct mysql protocol
r
Yes, StarRocks supports HTTP-based requests in addition to the standard MySQL protocol. 1. HTTP SQL API (Queries) Starting from v3.2.0, StarRocks introduced a native HTTP SQL API that allows you to execute queries without a MySQL client. * Supported Statements: 
SELECT
, 
SHOW
, 
EXPLAIN
, and 
KILL
. * Format: Requests are sent as JSON via POST, and results are returned in Newline Delimited JSON (ndjson) format. * Authentication: Uses Basic Auth (username/password). Example Request:
Copy code
bash
curl -X POST 'http://<FE_IP>:8030/api/v1/catalogs/default_catalog/databases/test/sql' \
-u 'root:password' \
-H 'Content-Type: application/json' \
-d '{"query": "SELECT * FROM my_table LIMIT 10;"}'
2. Other HTTP Interfaces Beyond running SQL queries, StarRocks uses HTTP for several other critical operations: * Data Ingestion (Stream Load): The primary way to load local files or data streams into StarRocks is via HTTP 
PUT
requests to the 
/api/{db}/{table}/_stream_load
endpoint. * Cluster Management: You can view cluster status, metrics (Prometheus format at 
/metrics
), and modify configurations dynamically via FE and BE HTTP interfaces. * Tableau/BI Support: While BI tools often use JDBC/ODBC (MySQL protocol), some web-based integrations leverage the HTTP API for lighter connectivity. When to use HTTP vs. MySQL Protocol * Use MySQL Protocol (Port 9030): For high-performance BI tools, CLI usage, complex application logic, and full SQL support (including DDL like 
CREATE
or 
ALTER
). * Use HTTP API (Port 8030): For lightweight health checks, simple data retrieval in web environments where a MySQL driver isn't available, or automated scripting where 
curl
is simpler. References • sql-reference/http_sql_api.md • administration/http_interface.md References • sql-reference/http_sql_api.md • administration/http_interface.md
k
@Rocky I have clients querying an external iceberg table like iceberg.table. I want to rename this table to be iceberg.tablev2. Does Starrocks support aliases so that when iceberg.table is being queried, it goes to iceberg.tablev2
r
Yes, StarRocks supports Logical Views, which are the standard way to create an alias for a table. If you rename your physical Iceberg table to 
tablev2
, you can create a view named 
table
that points to it, allowing your clients' existing queries to continue working without modification. Depending on how you have integrated Iceberg (via a Catalog or the External Table resource), here are the two ways to achieve this: 1. If using an External Catalog (Recommended) If you are using an Iceberg Catalog (e.g., 
CREATE EXTERNAL CATALOG iceberg ...
), StarRocks maps the remote tables 1:1. Once the table is renamed to 
tablev2
in your source metastore (Glue, Hive, etc.), it will appear as 
iceberg.db.tablev2
in StarRocks. To create an alias, you can create a StarRocks View. * Recommendation: Since v3.5, StarRocks supports creating Iceberg Views directly in the external catalog. This is ideal because the "alias" is then stored in your Iceberg catalog itself.
Copy code
sql
-- Create an Iceberg view as an alias
CREATE VIEW iceberg.db.table AS 
SELECT * FROM iceberg.db.tablev2;
_Note: If your version is older than v3.5, you must create the view inside a StarRocks internal database (in the 
default_catalog
)._ 2. If using External Tables (Legacy Resource Method) If you created a specific external table using the 
CREATE EXTERNAL TABLE
syntax (pointing to a 
RESOURCE
), you have even more flexibility. You can simply define the StarRocks table name independently of the physical Iceberg table name. If you want the StarRocks table 
table
to point to the remote Iceberg table `tablev2`:
Copy code
sql
CREATE EXTERNAL TABLE iceberg_db.table (
    id INT,
    data STRING
) ENGINE=ICEBERG 
PROPERTIES ( 
    "resource" = "iceberg_resource", 
    "database" = "remote_db", 
    "table" = "tablev2"  -- This points to the new physical name
);
Important Considerations * Metadata Refresh: If you rename the table in the source metastore (like Glue or Hive), you may need to run 
REFRESH EXTERNAL TABLE iceberg.db.tablev2
or 
REFRESH CATALOG iceberg
to ensure StarRocks sees the new name immediately. * Permissions: Ensure that the clients querying the new 
VIEW
have 
SELECT
permissions on both the view and the underlying 
tablev2
. * StarRocks Version: * v3.2+: Supports HTTP SQL API if your clients use HTTP. * v3.5+: Supports native Iceberg View creation. References • data_source/catalog/iceberg/DDL.md • sql-reference/sql-statements/View/CREATE_VIEW.md • data_source/External_table.md References • data_source/feature-support-data-lake-analytics.md • data_source/catalog/iceberg/DDL.md • sql-reference/sql-statements/View/CREATE_VIEW.md • data_source/External_table.md
k
@Rocky what is the query performance of maps that i will be adding to over tmime
r
The query performance of 
MAP
columns in StarRocks depends significantly on how you access the data and the volume of keys. As you add more keys over time, here is how the performance typically scales and how to optimize it: 1. Default Query Performance (Reading the "Blob") By default, StarRocks stores 
MAP
data (similar to 
JSON
and 
STRUCT
) as a collective item (a blob) within a row. * The Bottleneck: When you query a specific key using 
map_column['key']
, StarRocks must read the entire map for that row from disk, de-serialize it, and then look up the key. * Scaling Issue: As you add more keys, the "blob" size grows. This increases I/O overhead (reading larger data blocks) and CPU overhead (de-serializing more keys to find the one you need). Performance will degrade linearly with the average size of the map if you are scanning many rows. 2. Optimization: Generated Columns (Recommended) If you have specific keys that are queried frequently, you should use Generated Columns. This is the most effective way to maintain high performance as your maps grow. * How it works: You define a new column that automatically extracts a specific key from the map and stores it as a standard columnar type (e.g., 
INT
or 
VARCHAR
). * Performance: Queries against the generated column perform at native columnar speeds, bypassing the map de-serialization entirely. * Example:
Copy code
sql
    ALTER TABLE my_table ADD COLUMN user_id INT AS (my_map['user_id']);
3. StarRocks v3.3+ "Flat JSON/Subfield" Optimizations In recent versions (specifically v3.3 and later), StarRocks has introduced optimizations to accelerate semi-structured data: * Subfield Pruning: The Cost-Based Optimizer (CBO) can sometimes prune subfields so that the system only reads the necessary paths. While this is most mature for the 
JSON
type (via "Flat JSON"), it is increasingly applied to other complex types. * Inverted Indexes: You can build an Inverted Index on the map column. This allows StarRocks to quickly locate which rows contain a specific key or value without scanning the entire table. 4. Comparison to JSON * MAP is strictly typed (e.g., 
MAP<STRING, INT>
). This makes it slightly more efficient than raw 
JSON
because the value types are fixed and don't need type-inference during every query. * JSON is more flexible for "schema drift" (where values can be different types) but may have more overhead unless StarRocks' "Flat JSON" optimization kicks in. Best Practices for Growing Maps 1. Limit "Scan All" Queries: Avoid 
SELECT my_map
if you only need one key. Use 
my_map['key']
to allow the engine to attempt subfield optimizations. 2. Use Generated Columns for Hot Keys: If 80% of your queries target the same 5 keys, extract them. 3. Monitor Segment Size: As maps grow, your data segments will grow. Ensure your 
compaction
is healthy to keep read performance stable. 4. Avoid Sorting on Maps: MAP columns cannot be used as sorting keys or prefix indexes, so they won't benefit from range-based pruning (Zonemaps). References • sql-reference/data-types/semi_structured/Map.md • table_design/generated_columns.md • table_design/indexes/inverted_index.md References • sql-reference/data-types/semi_structured/Map.md • table_design/indexes/inverted_index.md • knowledge/usage/best-practice.md • best_practices/query_tuning/schema_tuning.md
d
@Rocky Is GRANT SELECT on an Iceberg view (external catalog, HMS) supported in StarRocks 3.5.20, without granting SELECT on its base tables?
r
In StarRocks v3.5.x (including 3.5.20), you can grant 
SELECT
privileges on an Iceberg view in an external catalog (HMS) without requiring the user to have 
SELECT
privileges on the underlying base tables. Key Details for v3.5.20 * Support Status: Starting with StarRocks v3.5, Iceberg views are fully supported for both creation and querying within Iceberg catalogs (Hive Metastore and REST). * Permission Model: StarRocks treats views as "secure" by default (since v3.4.1). This means the security mode defaults to 
SECURITY NONE
, where the system only checks the user's permission on the view itself, not the base tables. * StarRocks Dialect Requirement: For StarRocks to correctly handle permissions and parse the view, it is highly recommended that the view be created via StarRocks or updated to include a StarRocks dialect. If a view was created in another engine (like Spark) and StarRocks cannot parse its definition, the permission check may fail or behave inconsistently. How to Configure To allow a user to query an Iceberg view without base table access: 1. Ensure the user has 
USAGE
on the catalog:
Copy code
sql
    GRANT USAGE ON CATALOG iceberg_catalog TO ROLE analytic_role;
2. Grant 
SELECT
on the specific view:
Copy code
sql
    GRANT SELECT ON VIEW iceberg_catalog.db_name.my_iceberg_view TO ROLE analytic_role;
3. Verify the View Security Type: By default, views created in StarRocks use 
SECURITY NONE
. You can check the view definition. If you specifically want to enforce base table checks, you would have to explicitly set it to 
SECURITY INVOKER
during creation (though this syntax is primarily for internal views; external Iceberg views generally default to the view-only permission model once recognized by StarRocks). Important Constraints * View Creation: StarRocks only supports querying Iceberg views that it can parse. If you are using views created in other engines, you may need to use the 
ALTER VIEW ... ADD DIALECT
statement in StarRocks 3.5 to ensure compatibility. * Metadata Reporting: Note that in some 3.5 sub-versions, Iceberg views might incorrectly appear as 
BASE TABLE
in 
SHOW FULL TABLES
or 
information_schema
. Despite this display bug, they behave as views regarding permissions. References • data_source/feature-support-data-lake-analytics.md • sql-reference/sql-statements/View/CREATE_VIEW.md • data_source/catalog/iceberg/DDL.md • sql-reference/sql-statements/account-management/GRANT.md References • data_source/feature-support-data-lake-analytics.md • release_notes/release-3.5.md • data_source/catalog/iceberg/iceberg_rest_security.md • administration/user_privs/authorization/privilege_item.md • sql-reference/sql-statements/account-management/GRANT.md • administration/user_privs/authorization/privilege_faq.md • data_source/catalog/iceberg/DDL.md
Open in Slack
PreviousNext
Powered by