@Rocky how would I configure FE and CN nodes to talk to localstack for s3

To configure StarRocks FE and CN nodes to talk to LocalStack (or any S3-compatible storage), you need to set the endpoint, credentials, and typically enable path-style access. 1. Shared-Data Mode Configuration If you are deploying StarRocks in shared-data mode (where S3 is the primary storage), you configure these settings in

fe.conf

. CN nodes will receive most of this configuration from the FE, but they must be able to reach the LocalStack network endpoint. In `fe.conf`:

properties
run_mode = shared_data
cloud_native_storage_type = S3

# LocalStack endpoint (ensure CN nodes can resolve this address)
aws_s3_endpoint = <http://localstack:4566>
aws_s3_region = us-east-1

# LocalStack default credentials
aws_s3_access_key = test
aws_s3_secret_key = test

# The bucket and optional sub-path
aws_s3_path = your-bucket-name/data

# Must be false for S3-compatible storage
aws_s3_use_aws_sdk_default_behavior = false
aws_s3_use_instance_profile = false

# Recommended to ensure the volume is created on startup (v3.1+)
enable_load_volume_from_conf = true

In `cn.conf`: CN nodes primarily need a

storage_root_path

for local caching, but ensure they can resolve the

aws_s3_endpoint

defined in the FE. 2. General Storage Access (Storage Volumes) For accessing LocalStack for external tables or general file loading, it is recommended to use Storage Volumes (introduced in v3.1). This allows you to define the connection once and reuse it.

sql
CREATE STORAGE VOLUME localstack_volume
TYPE = S3
LOCATIONS = ("<s3://your-bucket-name/>")
PROPERTIES
(
    "aws.s3.endpoint" = "<http://localstack:4566>",
    "aws.s3.access_key" = "test",
    "aws.s3.secret_key" = "test",
    "aws.s3.enable_path_style_access" = "true",
    "aws.s3.enable_ssl" = "false",
    "aws.s3.use_aws_sdk_default_behavior" = "false"
);

-- Set as default if using shared-data
SET localstack_volume AS DEFAULT;

3. Key LocalStack Considerations * Path-Style Access: Most S3-compatible tools like LocalStack require

aws.s3.enable_path_style_access = true

. Without this, StarRocks may try to use virtual-hosted style requests (e.g.,

bucket.localstack:4566

), which often fails in local Docker environments. * SSL: Set

aws.s3.enable_ssl = false

if LocalStack is running on plain HTTP (port 4566). * Network Visibility: If StarRocks is in Docker and LocalStack is in another container, ensure they are on the same Docker network and use the container name (e.g., `http://localstack:4566`) as the endpoint. For more details, see the StarRocks documentation on S3-compatible storage. References • administration/management/FE_configuration.md