@Rocky

CREATE EXTERNAL CATALOG my_hive_catalog
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",
    -- Per-bucket credentials for bucket-a
    "hadoop.conf.fs.s3a.bucket.bucket-a.access.key" = "ACCESS_KEY_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.secret.key" = "SECRET_KEY_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>",
    -- Per-bucket credentials for bucket-b
    "hadoop.conf.fs.s3a.bucket.bucket-b.access.key" = "ACCESS_KEY_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.secret.key" = "SECRET_KEY_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>"
);

multibucketing in external catalog does not work? on 3.5.16 version

In StarRocks (including version 3.5.x), configuring per-bucket S3 credentials using the

hadoop.conf.fs.s3a.bucket.<bucket-name>...

syntax inside the

CREATE EXTERNAL CATALOG

properties does not work for data scanning. While this syntax is standard for the Hadoop S3A connector (Java), StarRocks uses a native C++ S3 client on the Backend (BE) nodes to perform high-performance data scans (e.g., for Parquet or ORC files). The native BE client does not automatically parse or support Hadoop's per-bucket configuration keys passed via the

hadoop.conf.

prefix. Key Reasons and Limitations 1. FE vs. BE Configuration: The

hadoop.conf.

prefix is used to inject properties into the Java-based Hadoop

Configuration

object on the Frontend (FE). This helps the FE's metadata client (like the Hive Metastore client) but is not natively propagated to the C++ S3 client on the BE for actual data reading. 2. Single Credential per Catalog: Most StarRocks external catalogs currently expect a single set of credentials (access key/secret key or IAM role) that applies to all buckets accessed within that specific catalog. Recommended Solutions 1. Use an IAM Role or Instance Profile (Recommended) The most robust way to handle multiple buckets in a single catalog is to use Instance Profile or Assumed Role authentication. Attach a policy to the IAM role that grants access to all required buckets (

bucket-a

,

bucket-b

, etc.).

sql
CREATE EXTERNAL CATALOG my_hive_catalog
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.region" = "us-west-2"
);

2. Use Separate Catalogs If the buckets belong to entirely different AWS accounts or require strictly separate IAM users, you should create separate catalogs in StarRocks, even if they point to the same Hive Metastore.

sql
-- Catalog for bucket-a
CREATE EXTERNAL CATALOG hive_bucket_a
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",
    "aws.s3.access_key" = "ACCESS_KEY_A",
    "aws.s3.secret_key" = "SECRET_KEY_A",
    "aws.s3.region" = "us-west-2"
);

-- Catalog for bucket-b
CREATE EXTERNAL CATALOG hive_bucket_b
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",
    "aws.s3.access_key" = "ACCESS_KEY_B",
    "aws.s3.secret_key" = "SECRET_KEY_B",
    "aws.s3.region" = "us-west-2"
);

3. Storage Volumes (Version 3.1+) StarRocks introduced Storage Volumes to manage credentials for different storage locations independently. While primarily used for Internal/Shared-Data tables, you can explore mapping specific storage volumes if your architecture allows, though for a standard Hive Catalog, the catalog-level credentials remain the primary method. If you specifically need the C++ client to support per-bucket credentials within a single catalog, this is currently a known limitation. You may want to follow the StarRocks GitHub for updates on multi-credential support for external catalogs. References • data_source/catalog/unified_catalog.md • administration/management/FE_configuration.md • data_source/catalog/hive_catalog.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md