@Rocky Does a StarRocks cluster maintain a persistent, always-on connection to the Hive Metastore (HMS) while it is running, or does it connect on-demand?

StarRocks does not maintain a single, "always-on" persistent connection to the Hive Metastore (HMS). Instead, it uses a connection pool and connects on-demand based on metadata needs. Key details of how this interaction works: 1. Connection Pooling StarRocks (specifically the Frontend, or FE) uses a Thrift connection pool to communicate with the HMS. * On-Demand: When a query or a background task needs metadata, it requests a connection from the pool. If a valid idle connection exists, it is reused; otherwise, a new one is created. * Timeouts: Connections are not permanent. They are subject to timeouts like

hive_meta_store_timeout_s

(default 10 seconds). There are also internal limits on how long a connection can stay in the pool before being retired. 2. Background Activity (Simulated "Always-On") While the network connections themselves are not "always-on," StarRocks can be configured to perform periodic background polling, which makes the interaction feel continuous: * `enable_background_refresh_connector_metadata`: When set to

true

(default in v3.0+), StarRocks polls the HMS at regular intervals (default every 10 minutes) to refresh cached metadata for frequently accessed catalogs. * Lazy Refresh: If background refresh is disabled, StarRocks only connects to the HMS when a query is executed and the cached metadata has expired (the "lazy update" policy). 3. Metadata Caching Because StarRocks caches Hive metadata (table schemas, partition locations, etc.), it often avoids connecting to the HMS entirely for repeated queries until the cache TTL (default 24 hours) or refresh interval (default 60 seconds) is reached. Summary: The connection is on-demand and pooled. You will see connections being opened and closed as needed by the FE, rather than one permanent pipe that stays open for the entire life of the StarRocks process. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • data_source/catalog/hive_catalog.md

@Rocky how can i check amount of http request from starrocks to hms server. can you tell me some metric of prometheus to find out which cluster creating that request?

To check the volume of requests from StarRocks to your Hive Metastore (HMS), you should look at Thrift-related metrics on the Frontend (FE) nodes. While you asked about "HTTP requests," StarRocks typically communicates with a standard Hive Metastore via the Thrift protocol (usually on port 9083). If you are using a specialized catalog like the Iceberg REST catalog, it uses HTTP, but standard Hive catalogs use Thrift. 1. Prometheus Metrics for HMS Requests Since HMS connections are managed through a Thrift client pool in the FE, you can monitor the activity using these metrics: *

thrift_used_clients

: The number of Thrift clients currently in use. A spike here often indicates heavy metadata activity (e.g., refreshing a large number of partitions). *

thrift_opened_clients

: The total number of open Thrift connections in the pool. *

thrift_connections_total

: A cumulative counter of all Thrift connections opened. You can use the

rate()

function in Prometheus to see the "requests per second" (e.g.,

rate(thrift_connections_total[5m])

). 2. Identifying the Requesting Cluster In Prometheus, clusters are typically distinguished by labels defined in your scrape configuration (

prometheus.yml

). To find out which cluster is making the requests, use the following labels: *

job

: Usually set to the cluster name or service name (e.g.,

job="starrocks-prod"

). *

instance

: Identifies the specific FE node IP/hostname making the request. *

group

: Commonly used in StarRocks Grafana templates to distinguish between

fe

and

be

. Example PromQL Query: To see the rate of new Thrift connections (requests) per cluster:

promql
sum(rate(thrift_connections_total[5m])) by (job, group)

3. Advanced Monitoring (Table Level) By default, the

/metrics

endpoint provides minified results. If you need to see if specific external metadata requests are causing pressure, you can request detailed metrics by adding a parameter to the scrape URL (this requires ADMIN privileges): * URL:

http://<FE_IP>:8030/metrics?with_table_metrics=all

* This will expose more granular metrics, though for HMS specifically, the Thrift pool metrics remain the primary source for connection volume. Summary for HMS Troubleshooting If you see a high volume of requests to your HMS: 1. Check Background Refresh: StarRocks v3.0+ enables

enable_background_refresh_connector_metadata

by default (every 10 mins). You can increase

background_refresh_metadata_interval_millis

to reduce frequency. 2. Monitor FE Logs: Search

fe.log

for

HiveMetaStoreClient

or

ThriftConnectionPool

to see errors or specific metadata tasks being executed. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • data_source/catalog/hive_catalog.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/management/monitoring/metrics.md • data_source/feature-support-data-lake-analytics.md

@Rocky if a table has 20 columns and 100 partition. what happend on hive catalog and hms if i select all columns with all partitions ?

When you query a Hive table with 20 columns and 100 partitions (using

SELECT *

without filters) for the first time or after a cache expiration, the interaction between StarRocks and the Hive Metastore (HMS) follows a specific sequence of requests. Here is exactly what happens on the Hive Catalog and the HMS: 1. Metadata Requests to HMS The StarRocks Frontend (FE) performs the following Thrift calls to the HMS: * Table Schema Fetch (1 call): StarRocks requests the table definition to identify the 20 columns and their data types. * Partition List Fetch (1 call): It calls

get_partition_names

to retrieve the list of all 100 partition names (e.g.,

date=2023-01-01

, etc.). * Partition Metadata Fetch (1 call): Using the list of 100 names, StarRocks makes a batch call (typically

get_partitions_by_names

) to get the storage descriptors (physical locations) for all 100 partitions at once. Because the default

max_partitions_in_one_batch

is 4096, a 100-partition request is handled in a single HMS request. 2. File Listing (Storage Interaction) This is often the most intensive part. Once the FE has the partition locations from the HMS: * Listing Requests (up to 100 calls): The FE must know which files exist in those partitions to create "splits" for the BEs. It will typically issue one file-listing request per partition to the remote storage (HDFS

listFiles

or S3

ListObjects

). * Total Requests: For 100 partitions, you will see approximately 100 metadata requests to your storage layer, separate from the HMS Thrift calls. 3. StarRocks Processing & Caching * Planning: The FE takes the file list and the 20 columns to build a physical execution plan, dividing the work among the Backends (BEs). * Caching: All the information (schema, partition locations, and file lists) is cached in the FE memory. * Subsequent Queries: If you run the same query again immediately, StarRocks will hit its internal cache and make zero requests to the HMS or Storage. * Background Refresh: If

enable_background_refresh_connector_metadata

is true (default in v3.0+), StarRocks will automatically refresh this cache every 10 minutes in the background, making occasional light requests to HMS to check for changes. Summary of "Load" Component | Requests (First Query) | Requests (Cached) Hive Metastore | ~3-5 Thrift calls (Batch) | 0 Storage (HDFS/S3) | ~100 listing calls | 0 FE Memory | Caches schema + 100 partition objects | Hits cache Tip: If you have thousands of partitions and see high latency, ensure

enable_metastore_cache

and

enable_remote_file_cache

are set to

true

(default) to avoid hitting the HMS and storage for every single query. References • data_source/catalog/hive_catalog.md • data_source/feature-support-data-lake-analytics.md • administration/management/FE_configuration.md References • introduction/Architecture.md • data_source/feature-support-data-lake-analytics.md

@Rocky When declare hive catalog with this. CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); if we put metastore uris as multiple values. what happend with connection count and load of that?

When you provide multiple comma-separated URIs in

hive.metastore.uris

, StarRocks implements a High Availability (HA) Failover strategy. It does not perform active, simultaneous load balancing across all URIs. Here is how it affects your connection count and load: 1. Connection Behavior: Random Failover StarRocks uses the standard Hive Thrift client logic for these URIs: * Selection: When the FE initializes or needs a new connection, it typically randomizes the list of URIs provided. * Stickiness: It picks one URI from the randomized list and attempts to connect. If successful, it "sticks" to that HMS instance for subsequent requests from that specific client/connection. * Failover: If the connection to the selected HMS fails (e.g., timeout or network error), the client will automatically try the next URI in the list. 2. Impact on Connection Count * No Multiplier Effect: Providing multiple URIs does not increase the total number of connections. If your FE is configured for a pool of 16 Thrift connections, it will still only maintain up to 16 connections total. * Concentration: In a healthy state, you will likely see the majority of connections from a single StarRocks FE concentrated on one of the HMS instances at a time, rather than a perfectly even 33/33/33% split. 3. Impact on HMS Load * Resource Usage: The total load (CPU/Memory) on your HMS cluster as a whole remains the same as if you had one URI, but it is distributed sequentially. * Isolation: If one HMS node becomes slow but doesn't completely "fail" (no socket error), StarRocks might stay connected to it, potentially causing query latency. To mitigate this, you can tune

thrift_rpc_timeout_ms

(default 10s) to force faster failover to a healthy node. Summary Table Feature | Behavior with Multiple URIs Strategy | Randomized Failover (Active-Passive per connection) Total Connections | Remains limited by FE Thrift pool settings (e.g.,

thrift_connections_total

) Distribution | Connections concentrate on the first "available" node reached Failover | Automatic; retries the next URI in the list upon

TTransportException

Recommendation: Use multiple URIs for reliability (HA). If you want to force even load balancing across your HMS nodes, you should put a dedicated hardware or software load balancer (like HAProxy) in front of the HMS nodes and provide that single LB VIP to StarRocks. References • data_source/catalog/hive_catalog.md • administration/management/FE_configuration.md • administration/management/resource_management/Load_balance.md • administration/management/monitoring/metrics.md References • data_source/catalog/hive_catalog.md • administration/management/FE_configuration.md • faq/Others.md • best_practices/bucketing.md • data_source/feature-support-data-lake-analytics.md