http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I need to create external catalog f...
# questions-and-troubleshooting
z
@Rocky I need to create external catalog for hive iceberg but I need to configure s3 creds per bucket and can I do it under one external catalog?
r
In StarRocks, external catalogs (including Iceberg) typically use a single set of storage credentials that apply to the entire catalog. There is currently no native property (like 
aws.s3.bucket.<name>.access_key
) to switch S3 credentials per bucket within one external catalog definition. If you need to access multiple buckets that require different credentials, you have three primary options: 1. Use an IAM Role with Multi-Bucket Permissions (Recommended) The most efficient way is to use Instance Profile or Assumed Role authentication. You can attach a policy to the IAM role that grants access to all required buckets. This allows a single catalog to query tables across different buckets without needing multiple sets of keys. * Example Property:
Copy code
sql
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.iam_role_arn" = "arn:aws:iam::123456789012:role/multi-bucket-access-role"
2. Create Multiple External Catalogs If the buckets must use strictly isolated IAM users (Access/Secret Keys), you should create a separate external catalog for each set of credentials. * Catalog A for 
bucket-1
using 
key-1
* Catalog B for 
bucket-2
using 
key-2
Even if they share the same Glue/Hive metastore, you can create two StarRocks catalogs pointing to the same metastore URI but with different S3 credentials. 3. Credential Vending (Iceberg REST Catalog) From StarRocks v4.0 onwards, the Iceberg REST catalog supports vended credentials. If you use a REST-based catalog (like Apache Polaris or Tabular), the REST server can provide temporary, scoped credentials (STS tokens) to StarRocks for each specific table or bucket. * Property: 
"iceberg.catalog.type" = "rest"
Summary of Properties When creating your catalog, these are the standard S3 credential parameters: Parameter | Description 
aws.s3.use_instance_profile
| Set to 
true
for Roles, 
false
for IAM Users. 
aws.s3.access_key
| IAM User Access Key. 
aws.s3.secret_key
| IAM User Secret Key. 
aws.s3.iam_role_arn
| ARN of the role to assume. 
aws.s3.region
| The AWS region (e.g., 
us-west-2
). Note: If you are using a Hive Metastore and have many buckets, ensure the IAM identity used by StarRocks has the 
s3:GetBucketLocation
, 
s3:ListBucket
, and 
s3:GetObject
permissions for every bucket path involved. References • data_source/catalog/hive_catalog.md • data_source/catalog/unified_catalog.md
Yes, you can configure S3 credentials on a per-bucket basis for Hive or Iceberg external catalogs in StarRocks, but there are specific ways to implement this depending on how you want to manage the configuration. Method 1: Using 
core-site.xml
(Recommended for BE/FE level) The most reliable way to handle per-bucket credentials within a single catalog is to use standard Hadoop S3A configurations. StarRocks' Hive and Iceberg connectors can respect properties defined in the Hadoop configuration files. 1. Add the bucket-specific properties to a 
core-site.xml
file:
Copy code
xml
    <configuration>
        <!-- Credentials for Bucket A -->
        <property>
            <name>fs.s3a.bucket.bucket-a.access.key</name>
            <value>ACCESS_KEY_A</value>
        </property>
        <property>
            <name>fs.s3a.bucket.bucket-a.secret.key</name>
            <value>SECRET_KEY_A</value>
        </property>

        <!-- Credentials for Bucket B -->
        <property>
            <name>fs.s3a.bucket.bucket-b.access.key</name>
            <value>ACCESS_KEY_B</value>
        </property>
        <property>
            <name>fs.s3a.bucket.bucket-b.secret.key</name>
            <value>SECRET_KEY_B</value>
        </property>
    </configuration>
2. Place this 
core-site.xml
in the 
fe/conf
directory of all FE nodes and the 
be/conf
directory of all BE nodes (or CN nodes). 3. Restart StarRocks. 4. Create your catalog without specifying global 
aws.s3
credentials (or they will act as the default fallback). Method 2: Pass properties in the 
CREATE EXTERNAL CATALOG
statement In recent versions of StarRocks, you can pass custom Hadoop configurations directly in the 
PROPERTIES
of the 
CREATE EXTERNAL CATALOG
statement by using the 
hadoop.conf.
prefix.
Copy code
sql
CREATE EXTERNAL CATALOG my_hive_catalog
PROPERTIES (
    "type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",
    -- Per-bucket credentials for bucket-a
    "hadoop.conf.fs.s3a.bucket.bucket-a.access.key" = "ACCESS_KEY_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.secret.key" = "SECRET_KEY_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>",
    -- Per-bucket credentials for bucket-b
    "hadoop.conf.fs.s3a.bucket.bucket-b.access.key" = "ACCESS_KEY_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.secret.key" = "SECRET_KEY_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>"
);
Important Considerations: * Native Client Compatibility: StarRocks BEs often use a native C++ S3 client for high-performance data reading. While the Java-based FE handles metadata using Hadoop properties, the native BE client primarily relies on the standard 
aws.s3.*
catalog properties. If the native client does not automatically map the 
fs.s3a.bucket
properties, you may encounter "Access Denied" errors during query execution even if 
SHOW TABLES
works. * Best Practice: The most stable approach is to use a single IAM Role (Instance Profile) or Assumed Role that has an IAM policy granting 
s3:GetObject
, 
s3:ListBucket
, etc., to all the relevant buckets. This avoids the complexity of managing multiple keys and ensures the native BE client has consistent access. * Storage Volumes: If you are using StarRocks v3.1+, you can also look into Storage Volumes, though they are typically used for internal/cloud-native storage and 
FILES()
functions rather than directly as the credential provider for an HMS-HMS based Hive catalog. References • administration/management/FE_configuration.md
z
@Rocky after what starrocks version s3a is supported?
r
Yes, you can configure S3 credentials on a per-bucket basis within a single external catalog (Hive or Iceberg) by leveraging Hadoop S3A properties. Supported Versions * v2.5+: General support for S3-compatible storage in external catalogs was introduced. At this stage, per-bucket configuration usually required manually adding 
core-site.xml
to your 
fe/conf
and 
be/conf
directories. * v3.1+: Added support for the 
hadoop.conf.
prefix in the 
PROPERTIES
of a 
CREATE EXTERNAL CATALOG
statement. This allows you to pass any Hadoop-level configuration (including S3A per-bucket keys) directly in the SQL command. * v3.2.4+: Introduced the 
fallback_to_hadoop_fs_list
property, which allows you to explicitly force StarRocks to use the Java-based S3A filesystem client for specific paths instead of the native C++ S3 client. How to configure it (v3.1 and later) You can use the 
hadoop.conf.fs.s3a.bucket.<bucket_name>.access.key
syntax within your catalog creation statement:
Copy code
sql
CREATE EXTERNAL CATALOG multi_bucket_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "<thrift://hms_host:9083>",
    
    -- Credentials for Bucket A
    "hadoop.conf.fs.s3a.bucket.bucket-a.access.key" = "AKIA_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.secret.key" = "SECRET_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>",
    
    -- Credentials for Bucket B
    "hadoop.conf.fs.s3a.bucket.bucket-b.access.key" = "AKIA_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.secret.key" = "SECRET_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.endpoint" = "<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>"
);
Important Usage Note By default, StarRocks BE (Backend) nodes use a native C++ S3 client for data reading to maximize performance. While the Java-based FE (Frontend) will correctly use the per-bucket S3A credentials to list metadata, the native BE client may require all buckets to be accessible via a single set of credentials (like an IAM Role/Instance Profile) or might not automatically map these specific S3A bucket properties. If you find that metadata loads (you can see tables) but queries fail with "Access Denied," you may need to force the use of the Hadoop S3A client for those specific buckets (available in v3.2.4+):
Copy code
sql
-- Add this to the PROPERTIES of your catalog
"fallback_to_hadoop_fs_list" = "<s3a://bucket-a/,s3a://bucket-b/>"
References • release_notes/release-2.5.md • deployment/shared_data/feature-support-shared-data.md • release_notes/release-3.4.md • release_notes/release-3.5.md • administration/management/FE_configuration.md • release_notes/release-3.2.md • release_notes/release-3.3.md
z
@Rocky I use 3.5.16 how can I configure there?
r
In StarRocks v3.5.16, you can configure per-bucket credentials within a single external catalog using the 
hadoop.conf.
prefix in your 
CREATE EXTERNAL CATALOG
statement. This version fully supports passing these Hadoop-level properties directly through SQL. Step 1: Create the Catalog with Per-Bucket Properties Use the 
hadoop.conf.fs.s3a.bucket.<bucket_name>.<property>
syntax. This allows the Java-based Frontend (FE) to resolve metadata for multiple buckets with different keys.
Copy code
sql
CREATE EXTERNAL CATALOG multi_bucket_iceberg
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",

    -- Credentials for Bucket A
    "hadoop.conf.fs.s3a.bucket.bucket-a.access.key" = "AKIA_BUCKET_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.secret.key" = "SECRET_BUCKET_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>",

    -- Credentials for Bucket B
    "hadoop.conf.fs.s3a.bucket.bucket-b.access.key" = "AKIA_BUCKET_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.secret.key" = "SECRET_BUCKET_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.endpoint" = "<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>"
);
Step 2: Configure the Backend (BE) for Data Reading By default, StarRocks' Backend (BE) uses a native C++ S3 client for high-performance reading. This native client does not always automatically pick up the specific 
hadoop.conf.fs.s3a.bucket
properties passed in the SQL statement. To ensure the BE can read the data from these different buckets, you have two options: Option A: Use a fallback (Recommended for v3.5.x) In your 
be.conf
file, you can specify that StarRocks should use the Hadoop Java-based S3A client (which respects the per-bucket keys) for these specific buckets: 1. Edit 
be.conf
on all BE nodes:
Copy code
properties
    fallback_to_hadoop_fs_list = <s3a://bucket-a/,s3a://bucket-b/>
2. Restart the BE nodes. Option B: Unified IAM Role (Preferred for Production) The most robust way in AWS is to attach an IAM Role to your BE/CN nodes that has permissions for both buckets. This eliminates the need for hardcoded keys in the SQL and allows the native C++ client to work at full speed. * Catalog property: 
"aws.s3.use_instance_profile" = "true"
Summary for 3.5.16 * Per-bucket configuration: Supported via 
hadoop.conf.fs.s3a.bucket...
properties in the catalog statement. * Client matching: If the native client fails to access the buckets, use 
fallback_to_hadoop_fs_list
in 
be.conf
to force the use of the S3A client for those paths. References • data_source/External_table.md • loading/s3.md • deployment/shared_data/s3.md • data_source/feature-support-data-lake-analytics.md
z
@Rocky it says [42000][1064] (Service: S3, Status Code: 404, Request ID: null) (Service: S3, Status Code: 404, Request ID: null)
r
The error 
[42000][1064] (Service: S3, Status Code: 404, Request ID: null)
in StarRocks typically indicates that the Frontend (FE) is unable to reach or find the Iceberg metadata files in S3. In StarRocks 3.5.16, a "Status Code: 404" with "Request ID: null" almost always points to an incorrect S3 endpoint or a region mismatch for the specific bucket you are trying to access. Because the Request ID is null, the request is likely failing on the client side (within the StarRocks FE Java code) before it even receives a response from a valid S3 server, or it's hitting an invalid URL that doesn't provide a Request ID. Steps to Resolve 1. Correct the Endpoint and Region When using per-bucket credentials, you must ensure that the endpoint for each bucket matches its physical AWS region. If 
bucket-a
is in 
us-west-2
and 
bucket-b
is in 
us-east-1
, they cannot share the same regional endpoint. Crucial: Do not include 
https://
in the 
hadoop.conf
endpoint properties unless specifically required by your S3-compatible provider. For AWS, use the domain only.
Copy code
sql
CREATE EXTERNAL CATALOG multi_bucket_iceberg
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "thrift://<hms_host>:9083",

    -- Bucket A configuration
    "hadoop.conf.fs.s3a.bucket.bucket-a.access.key" = "AKIA_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.secret.key" = "SECRET_A",
    "hadoop.conf.fs.s3a.bucket.bucket-a.endpoint" = "<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>",
    
    -- Bucket B configuration
    "hadoop.conf.fs.s3a.bucket.bucket-b.access.key" = "AKIA_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.secret.key" = "SECRET_B",
    "hadoop.conf.fs.s3a.bucket.bucket-b.endpoint" = "<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>"
);
2. Check for Path Style Access If you are using a MinIO or a custom S3 provider (not standard AWS), you might need to enable path-style access for the FE to find the metadata:
Copy code
sql
"hadoop.conf.fs.s3a.path.style.access" = "true"
3. Force Hadoop S3A (The "Fallback" trick) StarRocks has a native C++ client in the Backend (BE) that is very fast but sometimes strict about credentials. If the FE can see the tables but queries fail with 404/403, you need to tell the BE to "fall back" to the Java Hadoop client (which correctly reads your per-bucket 
hadoop.conf
properties). In your 
be/conf/be.conf
on all nodes: 1. Add: 
fallback_to_hadoop_fs_list = <s3a://bucket-a/,s3a://bucket-b/>
2. Restart the BE. Why you see "Request ID: null" * Wrong Endpoint: You might be providing 
<http://s3.amazonaws.com|s3.amazonaws.com>
(the global endpoint) for a bucket in a region that requires regional endpoints (like 
us-east-2
). * Network/DNS: The FE cannot resolve the endpoint name you provided. * Metadata Path Mismatch: If your Hive Metastore (HMS) has table locations starting with 
s3://
, but your properties are defined for 
s3a://
, StarRocks usually bridges this, but try to keep them consistent if possible. Summary Checklist for 3.5.16: 1. Ensure 
iceberg.catalog.type
is correct (usually 
hive
or 
glue
). 2. Verify that the bucket names in your 
hadoop.conf.fs.s3a.bucket.<name>
exactly match the bucket names in S3. 3. Ensure each bucket has its own 
.endpoint
property if they are in different regions. 4. Remove any 
http://
or 
https://
from the endpoint values. References • knowledge/usage/storage/hdfs_storage_volume_extended.md • administration/management/FE_configuration.md • deployment/shared_data/hdfs.md • administration/management/BE_configuration.md
z
@Rocky where can I check the exact error message? "
r
The error occurs because the property keys you used (
aws.s3a.bucket...
) are not recognized by StarRocks or the underlying Hadoop S3A client. For per-bucket credentials, you must use the 
hadoop.conf.
 prefix followed by the standard Hadoop S3A property names. In StarRocks v3.5.16, please update your catalog creation statement with the following corrections: 1. Use correct property names: Change 
aws.s3a.bucket...access_key
to 
hadoop.conf.fs.s3a.bucket...access.key
(note the 
hadoop.conf
prefix and the use of dots 
.
instead of underscores 
_
in 
access.key
). 2. Remove brackets: Remove the 
< >
from your Hive URI and endpoints. 3. Clean up endpoints: For the S3 endpoint, it is usually better to provide just the host and port (e.g., 
tt:443
) without 
https://
when using the S3A client. Corrected SQL
Copy code
sql
CREATE EXTERNAL CATALOG dataeng_iceberg_catalog_v2 
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "<thrift://hive-hive-metastore>",
    "enable_iceberg_metadata_cache" = "false",

    -- Bucket 1: dev-tbc-dlh
    "hadoop.conf.fs.s3a.bucket.dev-tbc-dlh.access.key" = "k",
    "hadoop.conf.fs.s3a.bucket.dev-tbc-dlh.secret.key" = "k",
    "hadoop.conf.fs.s3a.bucket.dev-tbc-dlh.endpoint" = "tt:443",
    "hadoop.conf.fs.s3a.bucket.dev-tbc-dlh.connection.ssl.enabled" = "true",
    "hadoop.conf.fs.s3a.bucket.dev-tbc-dlh.path.style.access" = "true",

    -- Bucket 2: uz-tbc-bank-dataplatform-development
    "hadoop.conf.fs.s3a.bucket.uz-tbc-bank-dataplatform-development.access.key" = "kk",
    "hadoop.conf.fs.s3a.bucket.uz-tbc-bank-dataplatform-development.secret.key" = "kk",
    "hadoop.conf.fs.s3a.bucket.uz-tbc-bank-dataplatform-development.endpoint" = "10.10.10.58",
    "hadoop.conf.fs.s3a.bucket.uz-tbc-bank-dataplatform-development.connection.ssl.enabled" = "false",
    "hadoop.conf.fs.s3a.bucket.uz-tbc-bank-dataplatform-development.path.style.access" = "true"
);
Important Next Step (Backend Configuration) The SQL above allows the Frontend (FE) to browse metadata. However, the Backend (BE) nodes use a native C++ client by default which does not support these per-bucket S3A properties. To ensure queries succeed, you must tell the BE nodes to use the Hadoop Java client for these buckets. 1. Edit 
be/conf/be.conf
on every BE node. 2. Add this line:
Copy code
properties
    fallback_to_hadoop_fs_list = <s3a://dev-tbc-dlh/,s3a://uz-tbc-bank-dataplatform-development/>
3. Restart your BE nodes. Summary of Property Key Changes Your current key | Corrected key 
aws.s3a.bucket.<name>.access_key
| 
hadoop.conf.fs.s3a.bucket.<name>.access.key
aws.s3a.bucket.<name>.secret_key
| 
hadoop.conf.fs.s3a.bucket.<name>.secret.key
aws.s3a.bucket.<name>.enable_path_style_access
| 
hadoop.conf.fs.s3a.bucket.<name>.path.style.access
aws.s3a.bucket.<name>.enable_ssl
| 
hadoop.conf.fs.s3a.bucket.<name>.connection.ssl.enabled
References • data_source/catalog/unified_catalog.md • administration/management/FE_configuration.md • data_source/External_table.md • integrations/authenticate_to_aws_resources.md • data_source/catalog/hive_catalog.md References • administration/management/FE_configuration.md • data_source/External_table.md • integrations/authenticate_to_aws_resources.md
To find the exact stack trace and details for this S3 404 error, you should check the logs on your StarRocks nodes. Since this error is happening during catalog interaction/metadata fetching, the Frontend (FE) logs are the most important. 1. Check the FE Logs (Priority) Go to the 
log
directory on your FE node (default path: 
starrocks/fe/log/
). * File: 
fe.log
or 
fe.warn.log
* What to look for: Search for the bucket name (
dev-tbc-dlh
) or the text 
Service: S3, Status Code: 404
. * Why: This will give you the full Java stack trace. It will often reveal the exact URL that StarRocks tried to hit (e.g., 
<https://dev-tbc-dlh.tt:443/>
vs `https://tt:443/dev-tbc-dlh/`), which helps determine if the issue is Path Style Access or a DNS/Endpoint configuration error. 2. Check the BE Logs (If queries fail) If the catalog is created successfully but a query fails, check the Backend (BE) nodes in 
starrocks/be/log/
. * File: 
<http://be.INFO|be.INFO>
or 
be.WARNING
* What to look for: Look for 
S3
or 
HDFS_FILE_READER
errors. If you are using the native client, the error might look different than the Java-based one in the FE. 3. Identify the Leader FE If you have multiple FEs, the metadata operations are handled by the Leader. Run this in your SQL client to find the leader:
Copy code
sql
SHOW FRONTENDS;
Look for the row where 
IsMaster
(or 
IsLeader
) is 
true
. Check the logs on that specific server. Common Cause for 404 in your specific config: In your previous SQL, you used 
"aws.s3a.bucket...endpoint" = "<https://tt:443>"
. * Protocol: The S3A client often prepends the bucket name to the endpoint. If it tried to reach 
dev-tbc-dlh.<https://tt:443>
, it would result in a 404 or a DNS failure. * Check the Log: The 
fe.log
will show the "Calculated Endpoint" or the actual URI being called. If you see a weird URL there, you know the configuration key was parsed incorrectly. Recommendation: After checking the log, ensure you have updated the keys to the 
hadoop.conf.fs.s3a...
format as suggested in the previous response, as the 
aws.s3a...
keys are likely being ignored entirely. References • administration/management/logs.md • administration/management/BE_configuration.md
z
@Rocky can I grep ?
r
Yes, you can use 
grep
directly on the StarRocks nodes. Since the error you're seeing involves a Java stack trace (on the FE) or C++ logs (on the BE), using context flags like 
-C
(to see lines before and after the match) is very helpful. 1. On the Frontend (FE) Leader node Run these from the 
starrocks/fe/log
directory: * Search for the specific error code:
Copy code
bash
    grep -C 10 "404" fe.log
* Search for your bucket name (to see the URL generated):
Copy code
bash
    grep -i "dev-tbc-dlh" fe.log | tail -n 20
* Search for the Request ID/S3 Service errors:
Copy code
bash
    grep -A 20 "Service: S3" fe.log
2. On the Backend (BE) nodes Run these from the 
starrocks/be/log
directory: * Search for S3-related failures:
Copy code
bash
    grep -i "s3" <http://be.INFO|be.INFO> | tail -n 50
* Look for 404 errors in the BE:
Copy code
bash
    grep "404" be.WARNING
What to look for in the 
grep
output: 1. The URI: Look for a line starting with 
GET
or 
HEAD
followed by a URL. If you see something like 
<https://bucketname.https://endpoint>
, it means your 
endpoint
property included 
https://
which caused a malformed URL. 2. Property Warnings: Look for lines saying 
Unknown configuration
or similar. If StarRocks didn't recognize your 
aws.s3a.bucket...
properties, it might have defaulted to standard AWS S3 logic, which would fail to find your private/custom S3 endpoint. 3. Cause by: In the FE Java stack trace, scroll down to the last 
Caused by:
section. That usually contains the most accurate reason (e.g., 
UnknownHostException
or 
SignatureDoesNotExit
). Tip: If the logs are rotating quickly, you can use 
zgrep
to search through the compressed 
.gz
log files in` files in the same directory.
Open in Slack
PreviousNext
Powered by