< Rocky> in fe configs what `iceberg metadata memory cache c StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> in fe configs what `iceberg_metadat...

RonJay

05/07/2026, 6:13 AM

@Rocky in fe configs what

iceberg_metadata_memory_cache_capacity

use for exactly?

Rocky

05/07/2026, 6:13 AM

In StarRocks, the FE configuration

iceberg_metadata_memory_cache_capacity

defines the maximum amount of memory (in bytes) dedicated to caching Iceberg table metadata on the Frontend (FE) node. Here is a breakdown of exactly what it does and why it is used: 1. Purpose: Accelerating Query Planning When you query an Iceberg table, StarRocks needs to read and parse metadata files (like manifest files) to determine which data files need to be scanned. This process can be slow because it involves remote storage I/O and CPU-intensive decompression/parsing of Avro files. This configuration allows StarRocks to keep these deserialized metadata objects in memory. Subsequent queries can skip the "fetch-decompress-parse" cycle, significantly reducing query planning time. 2. Part of a Two-Level Cache StarRocks uses a tiered caching strategy for Iceberg metadata: * Memory Cache (Level 1): Governed by

iceberg_metadata_memory_cache_capacity

. This is the fastest tier. * Disk Cache (Level 2): Governed by

iceberg_metadata_disk_cache_capacity

. If metadata is evicted from memory (LRU), it can be written to the local FE disk to avoid re-fetching from remote storage (S3/HDFS). 3. Key Technical Details * Default Value:

536870912

bytes (512 MB). * Algorithm: It uses the Least Recently Used (LRU) algorithm. When the 512 MB limit is reached, the oldest/least used metadata entries are evicted to make room for new ones. * What is stored: It primarily stores parsed manifest files (data file paths, partition stats, etc.) as memory objects. * Scope: This is a global FE configuration (set in

fe.conf

) and applies to all Iceberg catalogs unless specified otherwise. When should you tune it? * Increase it: If you have a large number of Iceberg tables or tables with a massive number of snapshots and manifest files, and you notice that query planning is consistently slow (high "PlanTime" in profile). * Decrease it: If your FE node is under severe memory pressure, although 512 MB is typically a safe and modest default for most production environments. Related Configs: * `iceberg_metadata_memory_cache_expiration_seconds`: How long an entry stays in memory (default is ~24 hours). * `enable_iceberg_metadata_disk_cache`: Whether to allow the second tier (disk) cache. References • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/BE_configuration.md • data_source/feature-support-data-lake-analytics.md

RonJay

05/07/2026, 6:18 AM

@Rocky How can starrocks prevent symptol of data stale when we using iceberg catalog and fe metadata disk? if there is icerberg metadata memory cache capcity = 0 all cached saved on disk? how they refresh with some options?

Rocky

05/07/2026, 6:18 AM

To prevent data staleness while maximizing the performance of your Iceberg catalog, StarRocks uses a multi-layered refresh and caching strategy. Here is exactly how it prevents staleness and what happens if you set the memory capacity to zero. 1. How StarRocks Prevents Data Staleness Even when using FE metadata disk caching, StarRocks avoids stale data through two primary mechanisms that track the Snapshot ID (the source of truth for Iceberg data freshness). * Metadata Pointer Refresh (
iceberg_table_cache_refresh_interval_sec
): StarRocks distinguishes between "Metadata Pointers" (which JSON file is the current snapshot) and "Metadata Content" (the manifest files). * By default, every 60 seconds, StarRocks checks if the metadata pointer is still fresh. * If a query occurs after this 60s window, StarRocks asynchronously (or sometimes synchronously) re-validates the latest Snapshot ID from the Metastore (Hive/Glue). If a new snapshot is found, the old cached metadata (even on disk) is marked as invalid for that table. * Background Periodic Polling: * enable_background_refresh_connector_metadata
: When

true

(default), the FE proactively polls the Metastore for changes every 10 minutes (configurable via

background_refresh_metadata_interval_millis

). * This ensures that even if no one is querying a table, the FE keeps the "Latest Snapshot ID" updated in memory. When a user finally runs a query, StarRocks already knows about the new data. 2. Setting
iceberg_metadata_memory_cache_capacity = 0
If you set the memory capacity to

, it does not simply "move" everything to the disk. Here is the technical behavior: * Execution requires Memory: StarRocks cannot "use" metadata directly from the disk. It must decompress and parse Avro manifest files into Java objects in the FE's heap memory to perform query planning (partition pruning, file filtering). * Constant Loading/Eviction: If capacity is

, StarRocks will fetch the metadata from the disk, load it into memory to plan the query, and because there is no capacity to "keep" it, the objects will be immediately eligible for Garbage Collection or eviction. * Result: You will experience significant performance degradation. Every single query will have to perform the "Disk I/O -> Decompress -> Parse" cycle because the memory "hot" layer is disabled. Recommendation: Do not set this to

. If you are worried about FE memory, set it to a small value like

128MB

256MB

rather than

. 3. How Refresh Works with Disk Cache When a refresh occurs (via the background thread or the 60s interval): 1. Metastore Check: StarRocks asks Hive/Glue: "What is the latest snapshot for Table X?" 2. ID Comparison: If the ID has changed (e.g., from

S1

S2

), the FE updates its Metadata Pointer Cache. 3. Cache Invalidation: The manifest files associated with

S1

(stored in memory or on disk) are no longer used for the "latest" version of the table. 4. New Fetch: The FE will fetch the new manifest files for

S2

. It will first look for them on disk (if

enable_iceberg_metadata_disk_cache

true

). If they aren't there, it fetches them from S3/HDFS, caches them to disk, and loads them into memory. Summary of Configs to Avoid Staleness If you want high freshness (low staleness) while using disk cache, tune these: Config | Recommended Value | Description

iceberg_table_cache_refresh_interval_sec

| Check for new snapshots more frequently.

background_refresh_metadata_interval_millis

(1 min) | Poll the metastore every minute instead of every 10.

enable_iceberg_metadata_disk_cache

true

| Keeps manifest files on FE local disk to speed up planning.

iceberg_metadata_memory_cache_capacity

> 0

| Keep at least some memory (512MB default) for parsed objects. References

Rocky

05/07/2026, 6:18 AM

• data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • administration/management/FE_configuration.md

RonJay

05/07/2026, 8:29 AM

@Rocky how can i set iceberg_table_cache_refresh_interval_sec in starrocks cluster? with session or fe.conf?

Rocky

05/07/2026, 8:29 AM

In StarRocks,

iceberg_table_cache_refresh_interval_sec

is a Catalog Property, not a setting for

fe.conf

or a session variable. Because it is a property of the catalog itself, you set it when you create the catalog or modify it using the

ALTER CATALOG

command. It was introduced in v3.5.7 to control the interval for asynchronous metadata refreshes. How to set it: 1. During Catalog Creation If you are setting up a new Iceberg catalog, include it in the

PROPERTIES

block:

Copy code

sql
CREATE EXTERNAL CATALOG iceberg_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "<thrift://localhost:9083>",
    -- Set the refresh interval here (e.g., 30 seconds)
    "iceberg_table_cache_refresh_interval_sec" = "30"
);

2. Modifying an Existing Catalog If the catalog already exists, use the

ALTER CATALOG

statement to update the property:

Copy code

sql
ALTER CATALOG iceberg_catalog SET PROPERTIES (
    "iceberg_table_cache_refresh_interval_sec" = "30"
);

Important Notes: * Version Requirement: This specific parameter is supported from v3.5.7 onwards. * Default Value: The default is

seconds. * Global Alternatives: If you want to control the background polling for all catalogs globally, you use the FE configuration

background_refresh_metadata_interval_millis

(which is set via

ADMIN SET FRONTEND CONFIG

). However,

iceberg_table_cache_refresh_interval_sec

is the preferred way to fine-tune specific Iceberg catalogs for faster data perception. * fe.conf vs. Catalog Props: Parameters related to memory/disk capacity (like the one you asked about earlier,

iceberg_metadata_memory_cache_capacity

) are set in

fe.conf

, but parameters related to refresh logic of specific data sources are almost always Catalog Properties. References • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/sql-statements/cluster-management/catalog/ALTER_CATALOG.md • data_source/feature-support-data-lake-analytics.md References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/deltalake_catalog.md

RonJay

05/07/2026, 11:18 AM

@Rocky how can i extend query timeout secend useing fe cn configuration?

Rocky

05/07/2026, 11:18 AM

To extend the query timeout in StarRocks, you primarily use session variables (which can be set globally to act as a system-wide configuration) or specific FE/CN configurations. Here are the different ways to achieve this: 1. Using Session Variables (Most Common) For standard SQL queries, the

query_timeout

variable is the primary control. The default is usually 300 seconds (5 minutes). * For the current session:

Copy code

sql
    SET query_timeout = 3600; -- Set to 1 hour for this session

* Globally (for all future sessions):

Copy code

sql
    SET GLOBAL query_timeout = 3600;

* For a single query (using a hint):

Copy code

sql
    SELECT /_+ SET_VAR(query_timeout = 3600) _/ * FROM large_table;

2. FE Configuration (
fe.conf
) If you want to change the default behavior or specific task timeouts at the cluster level, you can modify

fe.conf

(or use

ADMIN SET FRONTEND CONFIG

for mutable parameters). * task_runs_timeout_second
: Sets the default timeout for TaskRuns (used in StarRocks Tasks/Scheduled jobs). _ _Default:* 14400 (4 hours). * brpc_send_plan_fragment_timeout_ms
: If you are seeing RPC-related timeouts when the FE sends plan fragments to CN/BE nodes. _ _Default:* 60000 (60 seconds). To set these dynamically without restarting:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("brpc_send_plan_fragment_timeout_ms" = "120000");

3. CN/BE Configuration (
cn.conf
or
be.conf
) Compute Nodes (CN) and Backend nodes (BE) generally follow the timeout instructed by the FE. However, there are underlying network/streaming timeouts you might need to adjust in extreme cases: * streaming_load_rpc_max_alive_time_sec
: Specifically for load-related timeouts on the backend. * tablet_writer_ignore_eagain_timeout_second
: Used during high-concurrency writes. 4. For Specific Operations * Insert/Load Timeout: If your

INSERT INTO ... SELECT

is timing out, use:

Copy code

sql
    SET insert_timeout = 3600;

* Connection Timeout: If the client connection is dropping before the query finishes:

Copy code

sql
    SET GLOBAL wait_timeout = 28800; -- Default 8 hours

Summary Recommendation If your goal is simply to allow longer-running SELECT queries, the best approach is: 1. Run

SET GLOBAL query_timeout = 3600;

(This acts as a "configuration" change that persists). 2. If you are using StarRocks v3.3.3+, you can also pin this to a specific user:

Copy code

sql
    ALTER USER 'your_user' SET PROPERTIES ('session.query_timeout' = '3600');

References • sql-reference/System_variable.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/resource_management/Query_management.md • administration/management/resource_management/query_queues.md

RonJay

05/07/2026, 11:34 AM

@Rocky can you tell me when create external catalog with some property, is there any option for catalog query time out? is query timeout option should be only set up on fe conf?

Rocky

05/07/2026, 11:35 AM

When creating an external catalog in StarRocks, there is no specific property called

query_timeout

that you can set within the

PROPERTIES (...)

block of the

CREATE EXTERNAL CATALOG

statement for most catalog types (Iceberg, Hive, Hudi, etc.). Instead, query timeouts are managed via System Variables and FE Configurations, depending on what specific part of the process you want to control. 1. Where to set Query Timeout (The standard way) The primary way to control how long a query against an external catalog can run is through the query_timeout
session variable. This is NOT set in

fe.conf

, but rather via SQL. * Global change (for all users/queries):

Copy code

sql
    SET GLOBAL query_timeout = 3600; -- Set timeout to 1 hour

* Per-session change:

Copy code

sql
    SET query_timeout = 600; -- Set timeout to 10 minutes for this session only

2. Catalog-Specific "Timeout" Properties While there isn't a general "query timeout" in catalog properties, there are specific properties for metadata or connection management that can be set during catalog creation: * Iceberg Metadata Collection: If your Iceberg metadata is very large and the "planning" phase is timing out, use the system variable:

SET GLOBAL metadata_collect_query_timeout = 120;

(Introduced in v3.3.3) * JDBC Catalogs: For JDBC catalogs, you can often append timeout parameters to the

jdbc_uri

(e.g., `jdbc:mysql://host:port?socketTimeout=60000`). * Iceberg Refresh Interval: As mentioned earlier,

iceberg_table_cache_refresh_interval_sec

controls how often StarRocks checks for new data, but it doesn't limit the query duration itself. 3. FE Configurations (
fe.conf
) You should only use

fe.conf

(or

ADMIN SET FRONTEND CONFIG

) for internal or global system timeouts that affect the cluster's health, rather than individual SQL queries. * jdbc_query_timeout_ms
: (v3.5.13+) A global FE setting that sets the timeout for SQL queries executed through JDBC catalogs. * hive_meta_store_timeout_s
: Controls how long the FE waits for a response from the Hive Metastore. * iceberg_commit_queue_timeout_seconds
: (v4.1.0+) Specifically for write operations (commits) to Iceberg. Summary: Should you use
fe.conf
or Catalog Properties? Requirement | Method | Location Increase SQL query time |

SET GLOBAL query_timeout = X;

| SQL CLI / Session Iceberg metadata loading |

SET GLOBAL metadata_collect_query_timeout = X;

| SQL CLI / Session JDBC specific query limit |

jdbc_query_timeout_ms

fe.conf

(or ADMIN SET) Metastore connection |

hive_meta_store_timeout_s

fe.conf

Conclusion: For a standard "query timeout" when using an Iceberg or Hive catalog, you should use the query_timeout
system variable via SQL, not a catalog property or

fe.conf

. References • administration/management/FE_configuration.md • data_source/catalog/jdbc_catalog.md • administration/management/BE_configuration.md

RonJay

05/13/2026, 7:45 AM

@Rocky can you tell me difference between system_time_zone and time_zone?

Rocky

05/13/2026, 7:45 AM

In StarRocks, the primary difference between

system_time_zone

and

time_zone

lies in where the time is derived from and whether it can be modified. 1.
system_time_zone
(The Server Time) * Definition: This represents the time zone of the physical or virtual machine hosting the StarRocks Frontend (FE). * How it's set: It is automatically detected from the Operating System (OS) when the FE process starts. * Modifiability: It is read-only. You cannot change this variable within StarRocks. To change it, you would need to change the time zone of the server's OS and restart the FE. * Use case: It serves as a reference for the environment the cluster is running in. 2.
time_zone
(The SQL/Session Time) * Definition: This is the time zone used for all SQL-level calculations, data displays, and time-related functions (like

NOW()

CURTIME()

, or

FROM_UNIXTIME()

). * How it's set: It defaults to

Asia/Shanghai

. You can change it at the session level or globally. * Modifiability: Highly flexible. * Session-level:

SET time_zone = 'America/New_York';

* Global-level:

SET GLOBAL time_zone = '+08:00';

Asia/Shanghai

Changeable? | No | Yes Impact | Informational | Affects

NOW()

UNIX_TIMESTAMP()

, etc. Important Recommendation The StarRocks documentation recommends that before loading data, you should align your global

time_zone

with your

system_time_zone

(if they differ) to avoid issues where

DATE

DATETIME

types are misinterpreted during the import process.

Copy code

sql
-- Check current settings
SHOW VARIABLES LIKE '%time_zone%';

-- Example: Align global time zone to UTC
SET GLOBAL time_zone = 'UTC';

References • administration/management/timezone.md • sql-reference/System_variable.md • faq/Sql_faq.md References • sql-reference/System_variable.md • administration/management/timezone.md • faq/Sql_faq.md • sql-reference/sql-functions/date-time-functions/current_timezone.md • deployment/environment_configurations.md

RonJay

05/13/2026, 8:17 AM

@Rocky if i set variables with utc, can i set again time_zone with session level? it still no changed.

Rocky

05/13/2026, 8:17 AM

Yes, you can absolutely override a global

time_zone

setting with a session-level setting. In StarRocks, session variables always take precedence over global variables for the duration of that session. If it appears "no changed," it is likely due to one of the following reasons: 1. You are checking
system_time_zone
instead of
time_zone
* system_time_zone
: This is read-only and represents the server's OS time. It will never change regardless of what you set in SQL. * time_zone
: This is the variable that actually affects your queries and functions like

NOW()

. How to verify:

Copy code

sql
-- Check both session and global values
SELECT @@session.time_zone, @@global.time_zone;

-- Check if NOW() actually changed
SELECT NOW();

2.
SET GLOBAL
does not affect the current session If you ran

SET GLOBAL time_zone = 'UTC'

, your current session will still use its original time zone. Only newly created sessions (new connections) will pick up the 'UTC' global value. * To change the current session immediately, you must run:

SET time_zone = 'UTC';

(without the

GLOBAL

keyword). 3. Checking the wrong "Scope" If you set the session time zone and then run

SHOW GLOBAL VARIABLES LIKE 'time_zone';

, you will still see the old global value. You should check the session variable instead:

Copy code

sql
SET time_zone = 'Asia/Shanghai';
SHOW VARIABLES LIKE 'time_zone'; -- This shows the session value

4. Client/IDE behavior Some SQL clients or BI tools maintain a "connection pool." If the tool reuses a connection that was opened before you changed the global setting, it will still have the old time zone. * Solution: Close the connection/tool and reconnect, or explicitly run the session-level

SET

command at the start of your script. Correct sequence to test:

Copy code

sql
-- 1. Set global to UTC
SET GLOBAL time_zone = 'UTC';

-- 2. Current session still shows old value, so change it manually for now
SET time_zone = 'America/New_York';

-- 3. Verify
SELECT @@time_zone; -- Should show 'America/New_York'
SELECT NOW();       -- Should show New York time

If you are still seeing no change, please share the exact SQL commands you are running and how you are verifying the result. result (e.g., the output of

SHOW VARIABLES LIKE '%time_zone%'

). References • administration/management/timezone.md • sql-reference/System_variable.md • faq/Sql_faq.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • sql-reference/sql-functions/date-time-functions/current_timezone.md References • administration/management/timezone.md • sql-reference/System_variable.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md

RonJay

05/15/2026, 1:58 AM

@Rocky can you tell me how to scan iceberg metadata more faster when disable caching metadata on FE pod?

Rocky

05/15/2026, 1:58 AM

When you disable metadata caching on the FE (e.g., by setting

enable_iceberg_metadata_cache = false

iceberg_meta_cache_ttl_sec = 0

), StarRocks must fetch and parse the Iceberg manifest files for every query. This is computationally expensive for the FE. To speed up metadata scanning in this scenario, you should shift the workload from the FE to the BE/CN nodes using Distributed Planning. 1. Enable Distributed Plan Mode By default, StarRocks tries to decide whether to parse metadata on the FE or BEs. If your FE is a bottleneck (common in pod-based deployments with limited resources), force the metadata parsing to be distributed across your BE/CN nodes. * System Variable:

Copy code

sql
    SET GLOBAL plan_mode = 'distributed';

* auto
(Default): The system chooses based on table size. * `distributed`: Forces BE/CN nodes to read, decompress, and filter manifest files in parallel. This is significantly faster for large tables when the FE cache is off. 2. Disable Column Statistics Collection Parsing column-level statistics (min/max/null counts) from manifest files is one of the slowest parts of metadata scanning. If you don't strictly need these for cost-based optimization (CBO), turn them off to speed up the "collection" phase. * System Variable:

Copy code

sql
    SET GLOBAL enable_iceberg_column_statistics = false;

* When

false

, StarRocks only collects the total row count, which drastically reduces the amount of metadata data processed per query. 3. Increase Metadata Collection Timeout If your metadata is very large and you are using the distributed plan, the collection might hit a timeout before it finishes. * System Variable:

Copy code

sql
    SET GLOBAL metadata_collect_query_timeout = 120; -- Default is 60s

4. Optimize the "Planning" Profiling To see exactly where the time is being spent (e.g., whether it's waiting for S3/HDFS or busy parsing Avro files), enable the metadata profile:

Copy code

sql
SET enable_metadata_profile = true;
-- After running a query, check the profile

Summary of Recommended Settings If you cannot use the FE cache, apply these session or global variables to maximize performance: Variable | Recommended Value | Why? plan_mode
|

'distributed'

| Moves the work from FE to multiple BEs/CNs. enable_iceberg_column_statistics
|

false

| Skips heavy parsing of min/max values in manifests. metadata_collect_query_timeout
| `120`+ | Prevents timeouts during large-scale metadata retrieval. enable_metadata_profile
|

true

| Allows you to identify if the bottleneck is IO or CPU. Note: If your Iceberg table has a massive number of small manifest files, even a distributed plan will be slow. Consider using Iceberg's

rewriteManifests

action to compact them, which is the most effective way to speed up scanning regardless of caching. References • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/System_variable.md • data_source/feature-support-data-lake-analytics.md • sql-reference/sql-statements/data-definition/CREATE_ROUTINE_LOAD.md • administration/management/FE_configuration.md References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md

RonJay

05/15/2026, 2:49 AM

@Rocky To handle problems of data stale when starrocks meta enable, what kind of configuration will be suitable? If meta cache false and data cache true, data stale problem will be solved?

Rocky

05/15/2026, 2:49 AM

To address data staleness in StarRocks when querying external catalogs (like Iceberg), the Metadata Cache on the Frontend (FE) is the primary setting to adjust. Will "Meta Cache False + Data Cache True" solve the problem? Yes, this will solve the staleness problem for Iceberg. * Meta Cache (FE level): This caches the Iceberg snapshot and the list of data files. If this is

false

, StarRocks will fetch the latest snapshot and manifest files from the source (e.g., Glue/HMS/S3) for every query. This ensures the FE always knows exactly which files are current. * Data Cache (BE level): This caches the content of the data files. Because Iceberg uses immutable files (any update or delete creates new files with unique names), the Data Cache on the BE nodes will not cause staleness. When the FE gets the fresh metadata, it will point to new file paths that aren't in the BE cache yet, so the BE will fetch the new data from the source and cache it correctly. ────────── Recommended "Freshness" Configuration Disabling the metadata cache entirely can significantly slow down query planning. Instead of setting it to

false

, a more balanced approach is to use a short TTL or periodic background refresh. 1. Optimal Automated Setup (Low Staleness, High Performance) Use these FE configurations to keep the metadata fresh without the overhead of a full re-scan on every query: Parameter | Recommended Value | Description

iceberg_table_cache_refresh_interval_sec

| Metadata is considered "stale" after 5 seconds. The next query will trigger a refresh.

enable_background_refresh_connector_metadata

true

| Enables StarRocks to poll the source in the background.

background_refresh_metadata_interval_millis

| Refreshes the cache every 1 minute in the background. 2. Manual Refresh (For instant updates) If you know exactly when your ETL job finishes, you can keep the cache enabled for performance and manually force an update:

Copy code

sql
REFRESH EXTERNAL TABLE your_iceberg_table;

3. Data Cache (BE) Best Practices If you are using the Data Cache (

enable_scan_datacache = true

), ensure your Iceberg writers are following standard practices (not overwriting existing files with the same name). If your source system somehow overwrites files with the same name, size, and modification time, the Data Cache could return stale results. However, this is virtually impossible with standard Iceberg. Summary * Staleness is almost always a Metadata Cache issue, not a Data Cache issue. * Setting metadata cache to false solves staleness but hurts performance. * The best middle ground is setting a low

iceberg_table_cache_refresh_interval_sec

(e.g., 10s) and ensuring background refresh is active. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • data_source/datalake_faq.md • data_source/catalog/hive_catalog.md • administration/management/FE_configuration.md FE_configuration) References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • data_source/datalake_faq.md • data_source/catalog/hive_catalog.md • administration/management/FE_configuration.md

Rocky

05/15/2026, 2:49 AM

• data_source/catalog/deltalake_catalog.md • data_source/data_cache_troubleshooting.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • data_source/catalog/maxcompute_catalog.md -catalog) • sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md • sql-reference/System_variable.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md References • data_source/data_cache_troubleshooting.md • data_source/catalog/maxcompute_catalog.md • data_source/catalog/hive_catalog.md • data_source/datalake_faq.md • data_source/feature-support-data-lake-analytics.md

RonJay

05/15/2026, 5:14 AM

@Rocky When extracting an Iceberg timestamptz column via Impala -> pandas -> to_parquet, the resulting Parquet keeps the UTC timezone metadata (isAdjustedToUTC=true), but the same flow via StarRocks (Iceberg catalog) -> pandas -> to_parquet produces tz-naive timestamps with no timezone info in the Parquet file. I assume this is because StarRocks uses the MySQL wire protocol, which strips TZ metadata from DATETIME values before the Python driver hands them to pandas — could you confirm, and let me know if there is a recommended way (server option, driver, Arrow Flight, etc.) to preserve the timezone end-to-end? For now I am planning to advise users to apply tz_localize('UTC') in pandas before to_parquet as a workaround.

Rocky

05/15/2026, 5:15 AM

Yes, your assessment is correct. The behavior you're seeing is a direct result of how the MySQL Wire Protocol handles date and time types. Why it happens 1. Type Mapping: In StarRocks, Iceberg's

timestamptz

is mapped to the

DATETIME

type. 2. Protocol Limitation: The MySQL wire protocol transmits

DATETIME

values as binary or string packets representing the "local" time (governed by the StarRocks session variable

time_zone

). Critically, the protocol does not include a timezone offset in the data packet. 3. Driver Behavior: When a Python driver (like

PyMySQL

mysql-connector-python

) receives these packets, it constructs a "naive"

datetime.datetime

object because the protocol hasn't provided any metadata to prove the value is UTC, even if your StarRocks session is set to UTC. 4. Pandas/Parquet: When Pandas receives a naive datetime, it treats it as such. Writing to Parquet then results in a column without the

isAdjustedToUTC

flag. Recommended Solution: Arrow Flight SQL To preserve timezone metadata and rich type info end-to-end without manual localization in Python, you should use Arrow Flight SQL. StarRocks supports Arrow Flight SQL, which transfers data in the Apache Arrow format. Unlike the MySQL protocol, Arrow explicitly supports

Timestamp

types with timezone metadata (e.g.,

timestamp[ns, tz=UTC]

). * How it works: When StarRocks serves an Iceberg

timestamptz

via Arrow Flight, it can preserve the UTC "awareness" in the Arrow schema. * Python Implementation: You can use the

adbc_driver_flightsql

pyarrow.flight

libraries to query StarRocks.

Copy code

python
    import adbc_driver_flightsql.dbapi as flightsql

    with flightsql.connect(uri="grpc+tcp://<fe_host>:19090",
                           db_kwargs={"username": "root", "password": ""}) as conn:
        with conn.cursor() as cur:
            cur.execute("SELECT ts_column FROM iceberg_catalog.db.table")
            # This returns a pyarrow.Table or pandas.DataFrame
            # with the timezone metadata preserved.
            df = cur.fetch_df()

Alternative: Forcing "Aware" Objects in Python If you must stick with the MySQL protocol, you can configure some drivers to return timezone-aware objects, though this is often machine-dependent. For

mysql-connector-python

, you can sometimes use the

raw=False

and specific

converter

options, but the tz_localize('UTC')
workaround you planned is actually the most reliable way to handle the MySQL protocol's shortcomings in a Pandas pipeline. Summary * Confirmation: Yes, the MySQL protocol strips TZ metadata because it only carries the "wall clock" time. * Best Fix: Use Arrow Flight SQL (available in StarRocks v3.2+) for high-speed, type-safe data extraction. * Workaround: Continue using

.dt.tz_localize('UTC')

in Pandas if you are restricted to the MySQL port (9030). References • unloading/arrow_flight.md • administration/management/timezone.md • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/System_variable.md • sql-reference/sql-functions/date-time-functions/to_datetime_ntz.md References • sql-reference/sql-functions/date-time-functions/to_datetime.md • administration/management/timezone.md • loading/Spark-connector-starrocks.md

Rocky

05/15/2026, 5:15 AM

• sql-reference/sql-functions/date-time-functions/to_datetime_ntz.md • faq/loading/Broker_load_faq.md • unloading/Spark_connector.md

RonJay

05/15/2026, 8:01 AM

@Rocky How to enable isAdjustedToUTC when starrocks insert into files to s3 with parquet

Rocky

05/15/2026, 8:01 AM

To enable the

isAdjustedToUTC

property (also known as Instant Semantics) when inserting data into S3 (or any remote storage) using Parquet format, you need to use specific Parquet writer properties in your

INSERT INTO FILES

EXPORT

statement. Recommended Configuration The property to control this is "parquet.is_adjusted_to_utc" = "true"
. Note that this property only takes effect when parquet.use_legacy_encoding
is set to false
(which is the default in recent versions), because legacy encoding (

INT96

) does not support Parquet logical types like

isAdjustedToUTC

. Example: INSERT INTO FILES

Copy code

sql
INSERT INTO FILES (
    "path" = "<s3://your-bucket/path/to/data/>",
    "format" = "parquet",
    "parquet.is_adjusted_to_utc" = "true",
    "parquet.use_legacy_encoding" = "false",
    "aws.s3.access_key" = "...",
    "aws.s3.secret_key" = "..."
)
SELECT * FROM your_local_table;

Example: EXPORT

Copy code

sql
EXPORT TABLE your_local_table TO "<s3://your-bucket/path/to/export/>"
PROPERTIES (
    "format" = "parquet",
    "parquet.is_adjusted_to_utc" = "true"
)
WITH BROKER (
    "aws.s3.access_key" = "...",
    "aws.s3.secret_key" = "..."
);

Key Behaviors to Note: 1. Instant Semantics (
true
): The system outputs a timestamp normalized to UTC. Each value unambiguously identifies a single instant on the timeline. StarRocks will use your current session's

time_zone

to convert the "naive" internal

DATETIME

value into a UTC timestamp for the Parquet file. 2. Local Semantics (
false
, Default): The system outputs a timestamp that represents the wall-clock time (year, month, day, etc.) without an offset. These values are displayed the same way regardless of the viewer's timezone but do not represent a specific instant on the global timeline. 3. Data Type Mapping: Internal StarRocks

DATETIME

columns will be mapped to the Parquet

TIMESTAMP

logical type with the

isAdjustedToUTC

flag set to the value you specify. 4. Version Support: This property is generally available in StarRocks v3.2 and later for

INSERT INTO FILES

. References • sql-reference/sql-functions/table-functions/files.md • loading/loading_introduction/feature-support-loading-and-unloading.md • unloading/INSERT_INTO_FILES.md • sql-reference/sql-statements/loading_unloading/EXPORT.md References • data_source/feature-support-data-lake-analytics.md • loading/loading_introduction/feature-support-loading-and-unloading.md • administration/management/timezone.md • faq/loading/Broker_load_faq.md • sql-reference/sql-functions/table-functions/files.md • loading/Spark-connector-starrocks.md • administration/management/BE_configuration.md

RonJay

05/18/2026, 12:27 AM

@Rocky give me some example insert in to file with minio should i set region?

Rocky

05/18/2026, 12:32 AM

When using

INSERT INTO FILES

with MinIO, there are a few specific properties you must set to ensure StarRocks communicates correctly with your local or private S3-compatible storage. 1. MinIO Example (INSERT INTO FILES) To export data from a StarRocks table to a Parquet file in MinIO:

Copy code

sql
INSERT INTO FILES (
    "path" = "<s3://my-bucket/export/data/>",
    "format" = "parquet",
    -- MinIO specific settings
    "aws.s3.endpoint" = "<http://10.0.0.1:9000>",   -- Your MinIO endpoint
    "aws.s3.enable_path_style_access" = "true",    -- Crucial for MinIO
    "aws.s3.enable_ssl" = "false",                 -- Set to true if using https
    "aws.s3.access_key" = "minioadmin",
    "aws.s3.secret_key" = "minioadmin",
    "aws.s3.region" = "us-east-1"                  -- See notes below
)
SELECT * FROM your_starrocks_table;

────────── 2. Should you set
region
? Yes, you should set it. Even though MinIO doesn't use AWS regions in the same way, the underlying AWS C++ SDK used by StarRocks often requires a non-empty region string to sign requests correctly (Signature Version 4). * What value to use: You can usually use any valid-looking string like

us-east-1

minio

. * Why it's needed: If omitted, some versions of the driver might default to a lookup that fails or results in a "Region is missing" error. ────────── 3. Key MinIO-Specific Configuration Tips Property | Value | Why? aws.s3.enable_path_style_access
| true
| Mandatory. AWS S3 uses virtual-host style (

<http://bucket.s3.amazonaws.com|bucket.s3.amazonaws.com>

), but MinIO typically uses path-style (

endpoint/bucket

). aws.s3.endpoint
|

http://...

| You must provide the full URI including the protocol and port. aws.s3.use_instance_profile
|

false

| Usually required when providing manual access/secret keys to prevent the driver from trying to find AWS IAM roles. Summary Checklist 1. Endpoint: Include the protocol (

http

https

). 2. Path Style: Set

aws.s3.enable_path_style_access

true

. 3. Region: Provide a dummy value (e.g.,

us-east-1

). 4. Permissions: Ensure the MinIO user has

s3:PutObject

and

s3:ListBucket

permissions on the target bucket. References • unloading/unload_using_insert_into_files.md • loading/minio.md • sql-reference/sql-functions/table-functions/files.md • data_source/catalog/iceberg/iceberg_catalog.md References • loading/minio.md • unloading/unload_using_insert_into_files.md • sql-reference/sql-functions/table-functions/files.md • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/hive_catalog.md • data_source/catalog/paimon_catalog.md • data_source/catalog/hudi_catalog.md • data_source/catalog/deltalake_catalog.md • loading/loading_introduction/feature-support-loading-and-unloading.md • administration/management/FE_configuration.md • sql-reference/sql-statements/loading_unloading/EXPORT.md

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

Rocky

05/18/2026, 12:32 AM

RonJay

05/18/2026, 2:57 AM

@Rocky hi, when i insert into files can i use parquet.is_adjust_to_utc true properfies for save files?

Rocky

05/18/2026, 2:58 AM

Yes, you can use the property "parquet.is_adjusted_to_utc" = "true"
when unloading data using

INSERT INTO FILES

. Correct Spelling and Usage Note that the parameter name includes the suffix "*ed*" (

is_adjusted_to_utc

). It must be used in the

PROPERTIES

section of your

INSERT INTO FILES

EXPORT

command. For this property to work correctly: 1. Version: Ensure you are on StarRocks v3.2 or later (which introduced

INSERT INTO FILES

for Parquet). 2. Legacy Encoding: Set "parquet.use_legacy_encoding" = "false"
(this is the default in recent versions). If legacy encoding is set to

true

, StarRocks uses

INT96

for timestamps, which does not support the

isAdjustedToUTC

logical flag. 3. Session Timezone: StarRocks will use your current session's

time_zone

to adjust the values into UTC before writing them to the Parquet file if this flag is enabled. Example

Copy code

sql
INSERT INTO FILES (
    "path" = "<s3://my-bucket/export/>",
    "format" = "parquet",
    "parquet.is_adjusted_to_utc" = "true",
    "parquet.use_legacy_encoding" = "false",
    "aws.s3.access_key" = "...",
    "aws.s3.secret_key" = "...",
    "aws.s3.region" = "us-east-1"
)
SELECT ts_column, other_cols 
FROM local_table;

Why this is needed By default, StarRocks

DATETIME

values are "naive" (they don't carry timezone info). Without this property, a Parquet writer might save them as local timestamps. Setting

is_adjusted_to_utc

true

tells downstream systems (like Spark, Trino, or Pandas) that the values in the file represent a specific point in time on the UTC timeline, enabling proper "Instant" semantics. References • sql-reference/sql-functions/table-functions/files.md • unloading/unload_using_insert_into_files.md • administration/management/timezone.md • loading/loading_introduction/feature-support-loading-and-unloading.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md References • sql-reference/sql-functions/table-functions/files.md • administration/management/BE_configuration.md • loading/loading_introduction/feature-support-loading-and-unloading.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • unloading/Spark_connector.md

Open in Slack

Previous Next