@Rocky SQL Error [1064] [42000]: BE access S3 file failed. starrocks version: 3.5.1-d550b36

The error SQL Error [1064] [42000]: BE access S3 file failed in StarRocks 3.5.1 typically indicates that the Backend (BE) nodes cannot reach or authenticate with your S3-compatible storage. Because you are using version 3.5.1, there are specific configuration requirements related to the transition to JDK 17 that may be causing this. 1. Check Version-Specific JVM Options (New in 3.5) StarRocks 3.5 requires JDK 17. If you are using an External Catalog (Hive, Iceberg, Hudi, Paimon) on S3, you must add specific "open" modules to your BE configuration to allow Java-based S3 clients to function correctly. In your

be/conf/be.conf

, ensure the following is added to `JAVA_OPTS`:

properties
JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf --add-opens=java.base/java.util=ALL-UNNAMED"

Note: If you use Java UDFs, you also need

--add-opens=java.base/java.nio=ALL-UNNAMED

and

--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

. 2. Inspect BE Logs for the Root Cause The SQL error 1064 is a general wrapper. To find the real issue (e.g., "Access Denied", "404 Not Found", or "Connection Timeout"), you must check the BE logs: 1. Go to the log directory of the BE node that executed the query (usually

be/log/be.INFO

or

be.WARNING

). 2. Search for the string

BE access S3 file failed

. 3. Look for the SdkResponseCode or SdkErrorMessage. • 403 Forbidden: Credentials (AK/SK) are incorrect or the IAM policy lacks

s3:GetObject

permissions. • 404 Not Found: The file path is incorrect or the S3 endpoint is misconfigured. • No response body/Connection Reset: Often a network or proxy issue between the BE and S3. 3. Verify S3 Configuration Parameters Ensure your S3 properties are correctly defined in your

FILES()

function or Catalog definition: * Endpoint: If using a non-AWS S3 (like MinIO), ensure the

aws.s3.endpoint

includes the protocol (e.g., `http://minio:9000`) and that

aws.s3.enable_path_style_access

is set to

true

. * Region: For AWS S3, verify

aws.s3.region

matches your bucket's region. * Credentials: If using IAM roles instead of AK/SK, ensure

aws.s3.use_instance_profile

or

aws.s3.use_aws_sdk_default_behavior

is enabled. 4. Common Troubleshooting Commands If you are using the

FILES()

table function, try a simple list to see if the BE can "see" the files at all:

sql
SELECT * FROM FILES(
    "path" = "<s3://your-bucket/your-file.parquet>",
    "format" = "parquet",
    "aws.s3.access_key" = "...",
    "aws.s3.secret_key" = "...",
    "aws.s3.region" = "..."
) LIMIT 1;

If the error persists: Please check the

<http://be.INFO|be.INFO>

log and provide the specific

SdkErrorMessage

or any stack trace found there for more precise help. References • faq/shared_data_faq.md • data_source/External_table.md • data_source/datalake_faq.md • faq/Sql_faq.md • faq/Deploy_faq.md • loading/s3.md

@Rocky these errors are rare and doesn't come often

If the "BE access S3 file failed" error occurs intermittently rather than every time, it is usually caused by transient network issues, S3 rate limiting, or resource timeouts rather than a permanent misconfiguration. Since you are on version 3.5.1, here are the most likely causes for "rare" failures and how to address them: 1. S3 Rate Limiting (Throttling) If you have a high-concurrency workload, S3 (or S3-compatible storage like MinIO) may occasionally return a

503 Slow Down

error. StarRocks' internal S3 client retries, but if the limit is consistently hit, it will eventually fail. * Action: Check your BE logs (

<http://be.INFO|be.INFO>

) for the string

SlowDown

or

LimitExceeded

. * Optimization: If you are querying many small files, consider merging them or increasing the S3 bucket's prefix performance. 2. Increase BE Object Storage Timeouts The default timeouts might be too aggressive for rare network spikes. You can increase these in your `be/conf/be.conf`:

properties
# Increase connection and request timeouts (values in ms)
# -1 uses the SDK default, setting explicitly can help with flaky networks
object_storage_connect_timeout_ms = 5000
object_storage_request_timeout_ms = 15000

Note: A restart of the BE is required for these changes to take effect. 3. DNS or Connection Pool Issues Intermittent failures can occur if the BE fails to resolve the S3 endpoint or if the connection pool is exhausted. * Check DNS: Ensure all BE nodes have stable access to your DNS server. A temporary DNS resolution failure will trigger this error. * TCP Wait: Check if the BE node has a high number of sockets in

TIME_WAIT

(using

netstat -n | awk '/^tcp/ {t[$NF]++} END {for (state in t) print state, t[state]}'

). 4. Clock Skew S3 authentication uses a timestamp. If a BE node's system clock drifts significantly (usually > 5 minutes) from the S3 service time, the request will fail. If NTP is not perfectly synced across your cluster, one "drifting" BE node could cause intermittent failures depending on which node handles the request. * Action: Run

date

on all BE nodes to ensure they are synchronized. 5. Identifying the specific "Rare" Cause Because this is intermittent, finding the exact stack trace is critical. When the error next occurs: 1. Run

show load

or check the query ID. 2. Search the

<http://be.INFO|be.INFO>

logs on all BE nodes for that timestamp or query ID. 3. Look specifically for the AWS SDK Error Code: * `RequestTimeout`: Network was too slow. * `SlowDown`: You are being throttled by S3. * `ExpiredToken`: If using IAM roles/STs, the token refresh failed. * `Connection reset by peer`: Usually a network/firewall middlebox issue. If you can provide the specific error message found in the BE logs, I can provide more targeted configuration values to tune the retry logic. References • administration/management/BE_configuration.md • loading/s3.md • faq/shared_data_faq.md References • administration/management/BE_configuration.md