@Rocky is there a way to define temp external catalog ? or one that will live thought the session only? The reason behind the question: We have test envs that are writing Iceberg data as local hadoop catalog and we want to be able to define temp catalog only for specific session to read from these hadoop local catalog

Currently, StarRocks does not have a native "Temporary External Catalog" feature that is automatically scoped to a single session. External catalogs are global objects persisted in the Frontend (FE) and require system-level privileges to create. However, here are the best ways to achieve your goal for test environments: 1. Manual Lifecycle Management (Recommended) Since StarRocks allows you to create and drop catalogs dynamically, you can simulate a session-level catalog by creating it at the start of your test and dropping it at the end.

sql
-- 1. Create the catalog for the session
CREATE EXTERNAL CATALOG temp_test_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hadoop",
    "iceberg.catalog.warehouse" = "<hdfs://nn:9000/test/warehouse>" -- or your local path
);

-- 2. Use it
SELECT * FROM temp_test_catalog.db.table;

-- 3. Cleanup at the end of the session
DROP CATALOG temp_test_catalog;

2. Use unique naming for Test Envs If you have multiple test environments running concurrently, you can define a unique catalog name per test run (e.g.,

test_env_123

,

test_env_456

) to avoid conflicts. 3. Note on "Temporary Tables" (v3.3+) StarRocks v3.3.1 introduced Temporary Tables, but these are strictly for the Internal Catalog (

default_catalog

) and the

olap

engine. They are used for storing intermediate session-based results and cannot be used to point to external Iceberg data sources. 4. Querying Iceberg without a Catalog? StarRocks primarily uses the Catalog abstraction for Iceberg because it needs to interact with the Iceberg metadata layer (snapshots, manifests). * The

FILES()

table function is designed for raw files (Parquet/ORC) and does not natively process Iceberg metadata. * The deprecated Iceberg External Table syntax exists but still creates a persistent metadata object in StarRocks. Summary of limitations * Privileges: Creating an external catalog requires the

CREATE EXTERNAL CATALOG

privilege. * Visibility: Once created, a catalog is visible to all sessions unless restricted by Fine-Grained Access Control (FGAC). * Cleanup: You must explicitly run

DROP CATALOG

to remove it; it will not disappear when the connection closes. If you are looking for a way to query data without any metadata persistence, StarRocks currently expects a Catalog or an External Table to be defined. References • data_source/catalog/iceberg/iceberg_catalog.md • quick_start/iceberg.md

@Rocky I don't see any mention of hadoop type catalog in documentation docs.starrocks.io/docs/…/iceberg_catalog#… Can you review this ?

You are correct that the official StarRocks documentation for Iceberg Catalogs primarily highlights Hive, Glue, REST, and JDBC as the standard values for

iceberg.catalog.type

. However, StarRocks does support the

hadoop

catalog type for Iceberg, even though it is often omitted from the main documentation tables (likely because the community and official support prioritize metastore-based catalogs for production consistency). Why it's not in the main list In Iceberg, a Hadoop Catalog uses the underlying file system (HDFS, S3, or local disk) to manage metadata through a directory structure rather than a dedicated database. * Production Note: StarRocks documentation often steers users toward Hive or REST because Hadoop catalogs are prone to race conditions on certain storage systems (like S3 without S3Guard) and do not support atomic renames as reliably as a real metastore. * Test Environments: For your use case (local test environments), the

hadoop

type is the correct way to point StarRocks to a directory-based Iceberg warehouse. How to define it You can create your "temporary" (session-managed) catalog using the following syntax:

sql
CREATE EXTERNAL CATALOG temp_iceberg_test
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hadoop",
    "iceberg.catalog.warehouse" = "<hdfs://nn:9000/path/to/warehouse>" -- or your local hdfs/s3 path
);

Important Considerations for Your Workflow 1. Manual Cleanup: As mentioned before, there is no

TEMPORARY

keyword for catalogs. You must run

DROP CATALOG temp_iceberg_test;

at the end of your test script. 2. HDFS/Storage Config: Since you are reading from a "local hadoop catalog," ensure the StarRocks Frontends (FE) and Backends (BE) have the necessary HDFS configurations (

core-site.xml

,

hdfs-site.xml

) or environment variables if the "local" path is actually an HDFS cluster. 3. Privileges: The user running the test must have

CREATE EXTERNAL CATALOG

permissions at the system level. Summary While the documentation focuses on

hive

and

rest

,

iceberg.catalog.type

=

hadoop

is functional in StarRocks and is the standard way to integrate with directory-based Iceberg catalogs used in many local test/dev environments. References • data_source/catalog/iceberg/iceberg_catalog.md (Shows standard types, but supports extended Iceberg properties) • Apache Iceberg: Hadoop Catalog (External Reference)) References • data_source/catalog/iceberg/iceberg_catalog.md