< Rocky> whats the endpoint in shared architechture when I C StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> whats the endpoint in shared archit...

Zurab Kiknavelidze

07/16/2026, 7:35 AM

@Rocky whats the endpoint in shared architechture when I CREATE TABLE and it goes into object store? /db/table/tablet?

Rocky

07/16/2026, 7:36 AM

In StarRocks shared-data architecture, table data is not stored using human-readable names like

/db/table/tablet

. Instead, it uses internal IDs (database ID, table ID, partition ID) to organize the directory structure in the object store. Directory Structure (v3.1.4 and later) The typical path follows this pattern:

<bucket_root>/<cluster_id>/<db_id>/<table_id>/<partition_id>/[data|meta]/

* data/
: Contains the physical segment data files (

.dat

). * meta/
: Contains the tablet metadata files. How to find the exact endpoint You can retrieve the specific object storage path for your table by running:

Copy code

sql
SHOW PROC '/dbs/<database_name>';

This will output a

StoragePath

column showing the root S3/GCS/Azure URI for that specific table. Key Details * Before v3.1.4: Data was stored in a flatter structure under a single directory per table. * From v3.1.4 onwards: Data is organized by Partition ID subdirectories to improve management and performance. * Tablets: In the shared-data (compute-storage separation) architecture, "tablets" are represented by metadata in the

meta/

folder and segment files in the

data/

folder, rather than having a dedicated

/tablet_id/

directory level in the object store itself. For more details, you can refer to the Shared-data FAQ. References • faq/shared_data_faq.md • introduction/Architecture.md • table_design/table_design.md • deployment/shared_data/s3.md • quick_start/shared-data.md • deployment/shared_data/minio.md

Zurab Kiknavelidze

07/16/2026, 7:37 AM

@Rocky well I have multiple partitions as it appears and when it is dropped only 1 partition is dropped others remains why?

Rocky

07/16/2026, 7:37 AM

In StarRocks shared-data architecture, when you drop a table or partition, the data is not always immediately removed from the object store due to several safety and architectural mechanisms. Here is why you might see some partitions remaining while others are gone: 1. The FE Recycle Bin (Catalog Trash) By default, a

DROP TABLE

command does not delete data immediately. It moves the table metadata to a "recycle bin" in the Frontend (FE). * Behavior: The data remains in object storage until the retention period expires. * Default Retention: Controlled by the FE configuration

catalog_trash_expire_second

(default is often 1 day). * Immediate Deletion: To bypass this and delete data immediately, you must use:

Copy code

sql
    DROP TABLE <table_name> FORCE;

2. The Vacuum Process (Garbage Collection) In shared-data mode, physical deletion of files from S3/GCS is handled by a background process called Vacuum. * Batching: The Vacuum process may process deletions in batches or be throttled to avoid hitting object storage rate limits. * Lag: If you have a large number of partitions or files, the vacuum tasks might be backlogged. You can check the vacuum status via internal FE metrics or logs. * Version Issues: In some older versions (specifically earlier than v3.1.8 or v3.2.3), there were known issues where automatic garbage collection could fail or stall, requiring a manual cleanup tool provided by StarRocks. 3. Expression Partitioning Behavior If you are using Expression Partitioning (introduced in v3.1), the behavior differs slightly: * When using

ALTER TABLE ... DROP PARTITION

, data for that specific partition is typically removed directly and cannot be recovered from the recycle bin. _ If you dropped the _entire table*, the recycle bin logic (Point 1) still applies to the whole table. 4. Shared Data vs. Shared Nothing In the architecture you are using (shared-data), the directory structure is

/db_id/table_id/partition_id/

. If you see "some partitions remain," it often means the Vacuum process has successfully reached and deleted some

partition_id

sub-directories but hasn't finished the rest of the list yet. How to verify/fix: 1. Check FE Config: Run

ADMIN SHOW FRONTEND CONFIG LIKE '%trash%';

to see your retention settings. 2. Check for Errors: Look at the FE leader's

fe.log

for any "LakeService" or "Vacuum" related errors. These often indicate permission issues or timeouts when trying to delete from the object store. 3. Force Drop: If you are testing and want everything gone now, use

DROP TABLE ... FORCE

. References • Shared-data FAQ • Shared-data Garbage Cleanup References • faq/shared_data_faq.md • table_design/data_distribution/expression_partitioning.md • knowledge/admin_and_operation/shared_data_compaction.md

Zurab Kiknavelidze

07/16/2026, 7:40 AM

@Rocky I create TEMPORARY tables and as it apperas at some cases and its mostly partitioned tables onlly 1 partition files are deleted others are kept

Rocky

07/16/2026, 7:40 AM

This behavior—where only the first partition's data is deleted while others remain—is often due to how the Vacuum (Garbage Collection) process handles session-based temporary tables in the shared-data architecture. In StarRocks v3.3.1 and later, Temporary Tables are session-bound. When a session ends (or if you manually run

DROP TEMPORARY TABLE

), StarRocks is supposed to trigger a cleanup of the metadata and the physical files in your object store (S3/GCS/Azure). Why partitions are being left behind: 1. Metadata vs. Physical Cleanup Delay: When a temporary table is dropped, the metadata is removed from the FE immediately. The physical files in the object store are then queued for the Vacuum process. If the vacuum task fails to iterate through all

<partition_id>

sub-directories correctly (due to a bug or timeout), it may only successfully clean the first directory it encounters. 2. Known Issues in v3.3/v3.4: There have been reported edge cases in early v3.3.x and v3.4.x versions where the cleanup logic for partitioned temporary tables in shared-data mode could fail to traverse the full list of partitions. Specifically, if the table was created using

CREATE TABLE AS SELECT

(CTAS) with dynamic partitioning, the vacuum process might miss partitions that weren't part of the initial metadata sync. 3. Vacuum Batching: The vacuum process typically processes deletions in batches. If the object storage has high latency or rate limiting, the vacuum might "time out" or hit a retry limit after the first few deletions, leaving the rest of the partitions for the next cycle (which might fail if the table's root metadata is already gone). How to verify and fix: * Check the FE Logs: Search your

fe.log

for keywords like

Vacuum

LakeService

, or

Delete

. Look for "failed to delete" or "RPC timeout" errors. This will tell you if the FE is trying to delete the files but failing. * Manual Cleanup Command: If you have leaked temporary tables, you can use a special command (introduced in v3.3.1) to force a cleanup of orphaned temporary tables:

Copy code

sql
    -- To clean up temporary tables from a specific session
    CLEAN TEMPORARY TABLE ON SESSION 'session_id';

    -- To check for existing temporary tables
    SELECT * FROM information_schema.temp_tables;

* Use FORCE Drop: If you are manually dropping these tables, try using the

FORCE

keyword to bypass the recycle bin and trigger immediate cleanup logic:

Copy code

sql
    DROP TEMPORARY TABLE <table_name> FORCE;

* Storage Path Check: Confirm the paths of the remaining files using:

Copy code

sql
    SHOW PROC '/dbs/<db_id>/<table_id>/partitions';

If the table is already dropped, you may need to manually delete the

/db_id/table_id/

directory from your bucket if the Vacuum process continues to skip it. Recommendation: If you are on an early version of v3.3, consider upgrading to the latest patch (e.g., v3.3.8+ or v3.4.2+) as several vacuum and metadata-sync bugs for shared-data tables were addressed in these releases. References • table_design/StarRocks_table_design.md • faq/shared_data_faq.md • sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md • release_notes/release-3.3.md References • table_design/StarRocks_table_design.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md

Zurab Kiknavelidze

07/16/2026, 7:43 AM

@Rocky I cant see information_schema.temp_tables in that table because its already dropped just some partitions are remain

Rocky

07/16/2026, 7:44 AM

If you have already dropped the table and its entry is gone from

information_schema

, you are likely dealing with orphaned data (data files in object storage that no longer have corresponding metadata in the Frontend). This typically happens in shared-data mode because the Vacuum process (the background cleaner) failed to finish its task or timed out before iterating through all partitions. Why only one partition was deleted In shared-data architecture, the Vacuum service iterates through partition directories one by one. If it encounters an error (like an S3/GCS timeout, a permissions issue, or a "Not Implemented" API error on certain object stores), it may stop or skip the rest of the table's folders. Because the table metadata is now gone, the standard StarRocks SQL commands cannot "see" these files to try again. How to clean up these "leaked" files 1. The Manual GC Tool (Recommended) StarRocks provides a specific command-line tool for this scenario. It scans your object storage, compares the files against the existing Tablet Meta, and identifies files that are no longer referenced. * Operation:

lake_datafile_gc

* Command Example:

Copy code

bash
    ./bin/meta_tool.sh --operation=lake_datafile_gc \
      --root_path="<s3://your-bucket/cluster_id/db_id/table_id/>" \
      --expired_sec=3600 \
      --conf_file=./conf/be.conf \
      --do_delete=true

_Note: Always run with

--do_delete=false

first to preview what will be deleted._ 2. Check for "Stale" Partition Threshold If your temporary table was created and dropped very quickly, the partitions might be caught by the

lake_autovacuum_stale_partition_threshold

setting. * Parameter:

lake_autovacuum_stale_partition_threshold

(Default: 12 hours) * Effect: If a partition hasn't been updated within this window, the system may skip it during AutoVacuum to save resources. You can try lowering this dynamically:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_stale_partition_threshold" = "1"); -- 1 hour

3. Inspect FE Logs for Deletion Errors Since you mentioned this happens specifically with partitioned temporary tables, look in your

fe.log

for: * `fail to delete`: This often shows the exact S3/GCS error. * `LakeService`: Search for RPC timeouts when calling the BE to perform the deletion. Prevention for Future Tables To ensure all partitions are wiped immediately and bypass the background vacuum queue, use the

FORCE

keyword when dropping temporary tables:

Copy code

sql
DROP TEMPORARY TABLE table_name FORCE;

If you are on a version earlier than v3.3.8 or v3.4.2, this is a known edge case where the vacuum worker could lose track of "multi-partition" cleanup tasks. Upgrading to a later patch version is highly recommended to improve the reliability of the Shared-data Vacuum. of the shared-data garbage collector. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/FE_configuration.md • release_notes/release-3.3.md References • knowledge/admin_and_operation/shared_data_compaction.md

Zurab Kiknavelidze

07/16/2026, 7:46 AM

@Rocky I have starrocks in k8s cluster how can I run manual gc?

Rocky

07/16/2026, 7:46 AM

To run manual garbage collection (GC) in a Kubernetes (K8s) cluster for StarRocks shared-data mode, you need to execute the lake_datafile_gc
tool from within one of your Compute Node (CN) or Backend (BE) pods. Since temporary tables and dropped partitions are managed by the background "Vacuum" process, running this tool manually forces a comparison between the metadata and the physical files in your object storage. Step 1: Access a CN/BE Pod Choose any running CN or BE pod in your cluster:

Copy code

bash
kubectl exec -it <your-cn-pod-name> -n <namespace> -- /bin/bash

Step 2: Locate or Download the Tool The tool is part of

meta_tool.sh

. In most modern StarRocks images, it is located in the

bin

directory. 1. Navigate to the directory:

Copy code

bash
    cd /opt/starrocks/be/bin/

2. If

meta_tool.sh

is missing (common in some slim images), you may need to download the standalone GC package provided by StarRocks:

Copy code

bash
    wget "<http://starrocks-release.oss-cn-zhangjiakou.aliyuncs.com/yxg%2Fdata_gc.tar.gz>" -O be.tar.gz
    tar -xzvf be.tar.gz
    cd be/bin/

Step 3: Configure Authentication The tool needs to talk to your S3/GCS bucket. If your K8s pods use IAM Roles (IRSA) or Instance Profiles, the tool should inherit those permissions automatically. If you use Access/Secret Keys, you must ensure they are in the

be.conf

file used by the tool: 1. Open

/opt/starrocks/be/conf/be.conf

. 2. Ensure these fields are present (or add them):

Copy code

properties
    object_storage_endpoint = <http://s3.amazonaws.com|s3.amazonaws.com>
    object_storage_access_key_id = <your-key>
    object_storage_secret_access_key = <your-secret>

Step 4: Run the Manual GC Run the command in Dry Run mode first to see what would be deleted. Replace the

root_path

with the base path of your database or table in S3. Dry Run (Safe):

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://your-bucket/starrocks_data/db_id/table_id/>" \
  --expired_sec=3600 \
  --conf_file=../conf/be.conf \
  --do_delete=false

Actual Deletion: Once you verify the list of files, change

do_delete

to `true`:

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://your-bucket/starrocks_data/db_id/table_id/>" \
  --expired_sec=3600 \
  --conf_file=../conf/be.conf \
  --do_delete=true

Why this happens with Temporary Tables (Troubleshooting) If this is happening frequently with temporary tables, your Vacuum process might be skipping them because they are considered "stale" or too new. You can try adjusting these FE configurations via SQL to make the automatic process more aggressive: 1. Reduce the Stale Threshold: By default, StarRocks may ignore partitions that haven't been touched in 12 hours during vacuum.

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_stale_partition_threshold" = "3600"); -- 1 hour

2. Increase Vacuum Parallelism:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_vacuum_parallel_tasks" = "16");

Note: Always use

DROP TEMPORARY TABLE <name> FORCE;

if you want to skip the recycle bin and trigger immediate cleanup in the future. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md References

Rocky

07/16/2026, 7:46 AM

• knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md

Zurab Kiknavelidze

07/16/2026, 7:51 AM

@Rocky can I run this manual gc on the whole bucket?

Rocky

07/16/2026, 7:51 AM

Yes, you can run the manual GC tool on a higher-level directory (like the cluster root prefix), but doing so on the entire bucket requires caution regarding performance and memory. Can you run it on the whole bucket? Technically, yes. The

root_path

parameter can point to the service directory where all your StarRocks data resides (e.g., `s3://your-bucket/starrocks_cluster_id/`). However, there are two major reasons why the official documentation recommends more granular paths (like database or table level): 1. Memory Consumption: The tool loads the "Tablet Meta" for every tablet found under the

root_path

into the memory of the pod where you run it. If you have millions of tablets across the whole bucket, the tool might crash with an Out of Memory (OOM) error or cause the pod to be killed by K8s. 2. Execution Time: The tool must scan every object in the bucket and compare it against the metadata. On a full-bucket scale, this can take many hours or even days depending on your data volume and S3 API rate limits. How to do it safely in K8s If you still want to run it on a large path, follow these safety steps: 1. Determine the "Service Directory" Don't just use

<s3://your-bucket/>

. Use the specific prefix StarRocks uses. You can find this by checking the storage volume:

Copy code

sql
-- Check the location of your storage volume
SHOW STORAGE VOLUMES;

Or check a specific database's path:

Copy code

sql
SHOW PROC '/dbs'; -- Get your <db_id>
SHOW PROC '/dbs/<db_id>'; -- Look at the 'StoragePath' column

The path usually looks like

<s3://bucket/cluster_id/db_id/table_id/>

. You would use

<s3://bucket/cluster_id/>

to cover the whole cluster. 2. Use a Long "Expired" Window To prevent accidental deletion of data from currently running transactions, set a safe buffer (e.g., 24 or 48 hours):

Copy code

bash
--expired_sec=172800  # 48 hours

3. Run in Dry Run Mode First This is mandatory for a large-scale cleanup.

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://your-bucket/your-prefix/>" \
  --do_delete=false \
  --audit_file=/tmp/gc_audit.txt

4. Monitor Memory in K8s Since you are running this inside a CN/BE pod, monitor the pod's memory usage:

Copy code

bash
kubectl top pod <pod-name>

If memory spikes near the pod's limit, you must stop the tool and run it on a more specific path (e.g., one database at a time). Summary Recommendation Instead of the whole bucket, I recommend running it on the Database level first. 1. Get the list of DB IDs from

SHOW PROC '/dbs'

. 2. Run the tool for each DB path:

<s3://bucket/cluster_id/<db_id>/>

. This significantly reduces the risk of OOM while still cleaning up all orphaned partitions for the tables you care about. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md

Rocky

07/16/2026, 7:51 AM

• knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/feature-support-shared-data.md References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/BE_configuration.md • deployment/shared_data/gcs.md • faq/shared_data_faq.md

Zurab Kiknavelidze

07/16/2026, 7:58 AM

@Rocky should I include storage_path in cn.conf?

Rocky

07/16/2026, 7:58 AM

The parameter in StarRocks shared-data mode (using CN nodes) is actually called storage_root_path
, not

storage_path

. Whether you should include it depends on what you are trying to do: 1. For Normal CN Operation (Local Cache) In a Kubernetes environment, storage_root_path
defines where the CN node stores its local data cache. * Yes, include it if you have persistent volumes (PVs) attached to your CN pods and want to speed up queries by caching data from S3/GCS locally. * Leave it empty (

storage_root_path =

) if your CN pods are ephemeral (no PVs) or if you want to disable local caching to save disk space. _ _Note: If disabled, every query will fetch data directly from object storage, which may increase latency.* 2. For Running the Manual GC Tool (
meta_tool.sh
) If you are specifically asking because you are setting up the

conf_file

to run the manual garbage collection tool: * No, the GC tool does not require
storage_root_path
. The tool doesn't use the local cache; it only needs to talk to your object storage. * What you MUST include in the config file for the GC tool are the object storage credentials (if not using IAM roles/IRSA):

Copy code

properties
    # Required for manual GC tool if not using IAM roles
    object_storage_endpoint = <http://s3.amazonaws.com|s3.amazonaws.com>
    object_storage_access_key_id = <your-ak>
    object_storage_secret_access_key = <your-sk>

Summary of Differences Parameter | Purpose | Cluster Mode storage_path
| Physical data storage (SST files) | Shared-Nothing (BE) storage_root_path
| Local metadata and Data Cache | Shared-Data (CN) Recommendation for K8s: Check your CN deployment (Helm chart or Operator spec). Usually, the StarRocks operator handles this automatically. If you are manually editing

cn.conf

to run the GC tool, just ensure the S3/GCS credentials are correct; the

storage_root_path

Rocky

07/16/2026, 7:58 AM

• deployment/shared_data/azure.md • deployment/shared_data/hdfs.md • faq/shared_data_faq.md • data_source/catalog/deltalake_catalog.md References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • deployment/shared_data/gcs.md • administration/management/FE_configuration.md • data_source/catalog/deltalake_catalog.md

Zurab Kiknavelidze

07/16/2026, 8:00 AM

@Rocky sorry I meant aws_s3_path

Rocky

07/16/2026, 8:01 AM

No, you do not need to include

aws_s3_path

cn.conf

. Here is the breakdown of why and where that parameter is actually used: 1.
aws_s3_path
is an FE-only parameter The

aws_s3_path

parameter is used in

fe.conf

during the initial setup of a shared-data cluster. It tells the Frontend (FE) where the default storage volume (the "root" of your data in S3) should be created. * CN Nodes: They do not use this parameter. They get the data locations dynamically from the FE when executing queries. * BE Nodes: Similarly, they do not require this in their

be.conf

. 2. For the Manual GC Tool (
meta_tool.sh
) When you run the manual garbage collection tool, it does not look for

aws_s3_path

in the config file. Instead, it uses the path you provide directly in the command line via the --root_path
argument. What you SHOULD include in
cn.conf
(or the tool's conf) for GC: The tool only needs the credentials and endpoint to talk to S3. If you aren't using IAM roles/IRSA, ensure these are in your config file:

Copy code

properties
aws_s3_endpoint = <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>
aws_s3_region = us-west-2
aws_s3_access_key = <your-access-key>
aws_s3_secret_key = <your-secret-key>

_(Note: Use the

aws_s3_

prefix for these credential parameters in newer versions, or the

object_storage_

prefix mentioned earlier; both are generally supported by the tool)._ Summary * In `cn.conf`: Only include

storage_root_path

(for the local cache) and your S3 credentials. * In the GC Command: Specify the bucket path using the

--root_path

flag. Example GC Command (no
aws_s3_path
in conf needed):

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://my-bucket/starrocks/db_12345/>" \
  --conf_file=../conf/cn.conf \
  --do_delete=false

References • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md References • administration/management/FE_configuration.md • deployment/shared_data/s3.md

Zurab Kiknavelidze

07/16/2026, 8:07 AM

@Rocky IO error: S3: fail to list s3://dev-starrocks-fe/meta: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Rocky

07/16/2026, 8:08 AM

The error PermanentRedirect
(HTTP 301) occurs because the S3 endpoint you have configured does not match the actual AWS region where your bucket resides. S3 buckets are region-specific. If your bucket is in

us-west-2

but you are using the global endpoint (

<http://s3.amazonaws.com|s3.amazonaws.com>

) or a different regional endpoint, AWS will reject the request with this redirect error. How to Fix It 1. Identify the Correct Region Check your AWS Console to find the exact region of the bucket

dev-starrocks-fe

. _ _Example:

us-east-1

eu-central-1

, etc.* 2. Update your Configuration File Open the configuration file you are passing to the manual GC tool (the one specified in

--conf_file=...

). You need to update both the endpoint and the region. If you are using the newer
object_storage_
prefix (common for the GC tool):

Copy code

properties
object_storage_endpoint = s3.<your-region>.<http://amazonaws.com|amazonaws.com>
# Some versions of the tool also look for:
aws_s3_region = <your-region>

If you are using the standard
aws_s3_
prefix:

Copy code

properties
aws_s3_endpoint = s3.<your-region>.<http://amazonaws.com|amazonaws.com>
aws_s3_region = <your-region>

Common Examples: * US East (N. Virginia):

<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>

* US West (Oregon):

<http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>

* Europe (Frankfurt):

<http://s3.eu-central-1.amazonaws.com|s3.eu-central-1.amazonaws.com>

3. (Optional) Use the Path-Style flag If you are using a non-AWS S3-compatible storage (like MinIO) or certain older AWS configurations, you might also need to ensure the endpoint includes the protocol:

Copy code

properties
aws_s3_endpoint = <https://s3>.<your-region>.<http://amazonaws.com|amazonaws.com>

Why the error "Unable to parse ExceptionName"? This is a known quirk of the C++ SDK used by StarRocks nodes. When S3 returns a 301 Redirect, the SDK sometimes fails to parse the XML error body correctly and prints "Unable to parse ExceptionName," but the "PermanentRedirect" message is the real indicator of the region mismatch. Verify and Rerun After updating the file, rerun your command. Ensure the

--root_path

also matches the bucket structure:

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://dev-starrocks-fe/meta/>" \
  --conf_file=../conf/cn.conf \
  --do_delete=false

References • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md • loading/s3.md References • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md • loading/s3.md • deployment/shared_data/gcs.md • administration/management/BE_configuration.md • faq/shared_data_faq.md • integrations/streaming/pipe/s3.mdx • administration/Meta_recovery.md • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md

Rocky

07/16/2026, 8:08 AM

• loading/s3.md • deployment/shared_data/gcs.md • administration/management/BE_configuration.md • faq/shared_data_faq.md • integrations/streaming/pipe/s3.mdx • administration/Meta_recovery.md References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md • deployment/shared_data/gcs.md • administration/management/BE_configuration.md

Zurab Kiknavelidze

07/16/2026, 8:31 AM

@Rocky IO error: S3: fail to list s3://dev-starrocks-fe/: curlCode: 6, Couldn't resolve host name

Rocky

07/16/2026, 8:31 AM

This error, curlCode: 6, Couldn't resolve host name
, means the machine running the tool cannot find the IP address for the S3 endpoint you provided. This is a DNS failure. Since you previously had a

PermanentRedirect

error, you likely updated the endpoint in your config file, but there is now a typo or a formatting error in that value. Check these 3 things in your configuration: 1. Verify the
aws_s3_endpoint
format Ensure the endpoint does not include

s3://

or trailing slashes. It should be a bare hostname. * Correct:

aws_s3_endpoint = <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>

* Incorrect:

aws_s3_endpoint = <s3://s3.us-west-2.amazonaws.com/>

(The

s3://

prefix belongs in your command's

--root_path

, not the config file). 2. Check for typos in the Region Double-check that the region string is correct. A single misplaced character will cause a DNS lookup failure. * Standard format:

s3.<region>.<http://amazonaws.com|amazonaws.com>

(e.g.,

<http://s3.us-east-1.amazonaws.com|s3.us-east-1.amazonaws.com>

). _ _Note: For some older regions like

us-east-1

, using

<http://s3.amazonaws.com|s3.amazonaws.com>

is common, but for newer regions, you must include the region string.* 3. Test DNS from the command line On the same machine where you are running

meta_tool.sh

, try to

ping

curl

the endpoint manually to see if your OS can resolve it:

Copy code

bash
# Try to resolve the hostname
nslookup <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>

# Or use curl to see if it connects
curl -I <https://s3.us-west-2.amazonaws.com>

If these fail with "Could not resolve host," the issue is with your machine's network settings or

/etc/resolv.conf

, not StarRocks. Example Correct Config for
meta_tool.sh
If your bucket is in

us-west-2

, your configuration file should look exactly like this:

Copy code

properties
# No s3:// here!
aws_s3_endpoint = <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>
aws_s3_region = us-west-2
aws_s3_access_key = <your-ak>
aws_s3_secret_key = <your-sk>

And your command should look like this:

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://dev-starrocks-fe/meta/>" \
  --conf_file=../conf/cn.conf \
  --do_delete=false

References • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md References • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • quick_start/shared-data.md • deployment/environment_configurations.md • faq/Others.md • administration/Meta_recovery.md • loading/s3.md • integrations/streaming/pipe/s3.mdx • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • quick_start/shared-data.md • deployment/environment_configurations.md • faq/Others.md • administration/Meta_recovery.md

Rocky

07/16/2026, 8:31 AM

Rocky

07/16/2026, 8:31 AM

• deployment/shared_data/azure.md • deployment/shared_data/hdfs.md • data_source/catalog/deltalake_catalog.md • administration/management/resource_group.md • sql-reference/sql-statements/data-definition/ALTER_ROUTINE_LOAD.md • administration/management/enable_fqdn.md • administration/management/FE_configuration.md • deployment/environment_configurations.md • administration/Meta_recovery.md • integrations/streaming/pipe/s3.mdx • administration/management/BE_configuration.md • deployment/shared_data/gcs.md • deployment/shared_data/minio.md • deployment/shared_data/azure.md • deployment/shared_data/hdfs.md • data_source/catalog/deltalake_catalog.md • administration/management/resource_group.md • sql-reference/sql-statements/data-definition/ALTER_ROUTINE_LOAD.md • administration/management/enable_fqdn.md References • integrations/streaming/pipe/s3.mdx • quick_start/shared-data.md • faq/Others.md • administration/Meta_recovery.md • faq/shared_data_faq.md • deployment/environment_configurations.md • loading/s3.md

Zurab Kiknavelidze

07/16/2026, 8:33 AM

@Rocky what properties should I have in cn.conf except object_storage_endpoint, object_storage_access_key_id, object_storage_secret_access_key?

Rocky

07/16/2026, 8:33 AM

Besides the credentials and endpoint, there are a few other critical properties you should include in your

cn.conf

(or

be.conf

if using the GC tool) to ensure proper connectivity and functionality in a shared-data environment. 1. The Region (Most Important for your previous error) Since you encountered a

PermanentRedirect

earlier, you must specify the AWS region. If you are using the

object_storage_

prefix, the corresponding property is: * object_storage_region = <your-region>
(e.g.,

us-west-2

) If that isn't recognized by your specific version of the tool, use the standard StarRocks S3 property: * aws_s3_region = <your-region>
2. Standard CN Properties If this is a configuration for an actual running Compute Node (and not just for a one-off tool), you need these: * starlet_port = 9070
: The heartbeat service port for shared-data clusters. * storage_root_path = <path>
: This defines where the local data cache is stored. Even though data is in S3, CNs cache "hot" data locally for performance. _ _Example:*

storage_root_path = /data/starrocks/storage

_ _Note:* If you want to disable local cache (not recommended for performance), you can leave this empty. 3. Storage Type While S3 is often the default, it is good practice to explicitly define the storage type: * cloud_native_storage_type = S3
4. Optional: AWS SDK Behavior If you are running on EC2 and want to use IAM Roles (IRSA/Instance Profiles) instead of hardcoded keys, you would remove the access/secret keys and add: * aws_s3_use_instance_profile = true
* aws_s3_use_aws_sdk_default_behavior = true
Recommended Minimal
cn.conf
for GC Tool If you are setting this up specifically to fix the "PermanentRedirect" and "Host not resolved" errors for the garbage collection tool, your file should look like this:

Copy code

properties
# S3 Connection
object_storage_endpoint = <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>
object_storage_region = us-west-2
object_storage_access_key_id = <your-ak>
object_storage_secret_access_key = <your-sk>

# CN Cache / Storage
storage_root_path = /opt/starrocks/cn/storage
starlet_port = 9070

Note on Property Names: In newer versions of StarRocks (3.0+), properties prefixed with

object_storage_

are often used for unified storage access, but the tool may still fall back to

aws_s3_

prefixes (

aws_s3_endpoint

aws_s3_access_key

aws_s3_secret_key

Rocky

07/16/2026, 8:33 AM

• administration/management/BE_configuration.md • administration/management/FE_configuration.md

Zurab Kiknavelidze

07/16/2026, 9:02 AM

@Rocky I can ping that dns from pod but that gc manual cleanup says IO error: S3: fail to list s3://dev-starrocks-fe/: curlCode: 6, Couldn't resolve host name

Rocky

07/16/2026, 9:06 AM

If you can

ping

the hostname from the pod but the tool still returns

curlCode: 6 (Couldn't resolve host name)

, the issue is likely not your network, but a formatting error in how the endpoint is written in your

cn.conf

(or

be.conf

) file that prevents the StarRocks S3 client (which uses

libcurl

) from parsing it correctly. Check for these specific formatting traps: 1. Remove the Protocol and Slashes The

object_storage_endpoint

(and

aws_s3_endpoint

) should generally be just the hostname. If you have

s3://

or a trailing

in the config file, the tool may fail to resolve it. * Wrong:

object_storage_endpoint = <https://s3.us-west-2.amazonaws.com/>

* Wrong:

object_storage_endpoint = <s3://s3.us-west-2.amazonaws.com>

* Correct:

object_storage_endpoint = <http://s3.us-west-2.amazonaws.com|s3.us-west-2.amazonaws.com>

2. Check for Invisible Whitespace Check your

cn.conf

for trailing spaces or tabs after the endpoint value.

ping

ignores these, but

libcurl

may include them in the DNS request, causing it to fail. * Fix: Ensure there is no space after the hostname before the newline. 3. Check for Proxy Environment Variables The GC tool uses

libcurl

, which respects system proxy variables (

http_proxy

https_proxy

no_proxy

ping

(ICMP) ignores these. If your Pod has a proxy configured that cannot reach the S3 endpoint,

ping

will succeed while the tool will fail. * Test: Run

echo $https_proxy

in your pod. If it's set, try running the tool with the proxy cleared:

Copy code

bash
    https_proxy="" http_proxy="" ./meta_tool.sh ...

4. Virtual-Hosted Style vs. Path Style In some environments (especially MinIO or certain VPC configurations), you may need to force a specific addressing style. If your endpoint is not a standard AWS one, try adding:

Copy code

properties
# Try adding this if using a non-standard or private S3 endpoint
aws_s3_use_virtual_hosted_style = false

5. Verify the Variable Name Ensure you aren't mixing up the prefixes. Depending on your StarRocks version, the tool might only be looking for one of these: * Try using

aws_s3_endpoint

instead of

object_storage_endpoint

(or vice versa) if the error persists. Recommendation: Test with
curl
directly To prove if it's a StarRocks config issue or a

curl

issue, run this from the same pod:

Copy code

bash
# Replace with your actual endpoint
curl -v <https://s3.us-west-2.amazonaws.com>

If this

curl

command works but the tool fails, the tool is likely misinterpreting a value in your

cn.conf

. Double-check that you are passing the correct

--conf_file

path to the

meta_tool.sh

command. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/s3.md • faq/shared_data_faq.md References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/s3.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • deployment/environment_configurations.md • loading/s3.md • integrations/streaming/pipe/s3.mdx • knowledge/admin_and_operation/shared_data_garbage_cleanup.md

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Rocky

07/16/2026, 9:06 AM

Zurab Kiknavelidze

07/16/2026, 9:08 AM

@Rocky its http not https

Rocky

07/16/2026, 9:09 AM

If you are using HTTP instead of HTTPS, and you can

ping

the host but the tool fails with curlCode: 6
(Couldn't resolve host name), the most likely cause is that the tool is trying to use Virtual-Hosted Style requests. In S3's default "Virtual-Hosted" style, the client tries to resolve a DNS name in the format:

bucket-name.endpoint-hostname

(e.g.,

<http://dev-starrocks-fe.your-s3-endpoint.com|dev-starrocks-fe.your-s3-endpoint.com>

) If your DNS server or your S3-compatible service (like MinIO) doesn't have a record for that specific subdomain, you get

curlCode: 6

. How to fix your
cn.conf
for HTTP: Add or update these properties in the config file you pass to the tool: 1. Remove the protocol from the endpoint: Ensure

object_storage_endpoint

(or

aws_s3_endpoint

) does not include

http://

. 2. Disable HTTPS: Explicitly tell the client to use HTTP. 3. Disable Virtual-Hosted Style: Use "Path-style" access (

endpoint/bucket

) which doesn't require extra DNS subdomains. Example
cn.conf

Copy code

properties
# 1. Use the bare hostname/IP and port
object_storage_endpoint = your-s3-host:9000 

# 2. Disable HTTPS (this forces HTTP/Port 80 logic)
aws_s3_use_https = false

# 3. Use Path-style access (CRITICAL for "Couldn't resolve host name" errors)
aws_s3_use_virtual_hosted_style = false

# Credentials
object_storage_access_key_id = <your-ak>
object_storage_secret_access_key = <your-sk>

Why
ping
works but the tool fails: * When you

ping your-s3-host

, your OS resolves the host correctly. * When the tool runs, it transforms the request into

dev-starrocks-fe.your-s3-host

. _ Unless you have a wildcard DNS record (

_.your-s3-host

), the lookup for the bucket-prefixed hostname will fail. Note: If the tool is still failing to recognize

object_storage_

prefixes, try replacing them with the older

aws_s3_

prefix counterparts: *

aws_s3_endpoint = your-s3-host:9000

aws_s3_access_key = <your-ak>

aws_s3_secret_key = <your-sk>

References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/s3.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • loading/s3.md References • deployment/shared_data/s3.md • administration/management/FE_configuration.md

Zurab Kiknavelidze

07/16/2026, 9:26 AM

@Rocky what if object sstorage only support path-style? what should I do? thats why it not resorvable

Rocky

07/16/2026, 9:26 AM

If your object storage only supports path-style access, the error

Couldn't resolve host name

is happening because StarRocks is defaulting to virtual-hosted style. In virtual-hosted style, the client tries to resolve

bucketname.endpoint

(e.g.,

<http://dev-starrocks-fe.your-s3-host.com|dev-starrocks-fe.your-s3-host.com>

). If your DNS doesn't have a record for that specific subdomain, it fails. To force path-style access (where it uses `your-s3-host.com/dev-starrocks-fe`) and ensure HTTP is used, add these properties to your configuration file: 1. The Fix for Path-Style & HTTP Add these exact lines to the configuration file you are passing to the tool (usually

cn.conf

or a standalone properties file):

Copy code

properties
# Force Path-style access (REQUIRED if your storage doesn't support bucket.host subdomains)
aws_s3_use_virtual_hosted_style = false

# Disable SSL to use HTTP (REQUIRED since you specified it's http)
aws_s3_use_https = false

2. Verify your Endpoint format Ensure your

object_storage_endpoint

(or

aws_s3_endpoint

) does not include the protocol (

http://

). The

aws_s3_use_https = false

flag handles the protocol transition. * Correct:

object_storage_endpoint = <http://your-s3-host.com:9000|your-s3-host.com:9000>

* Incorrect:

object_storage_endpoint = <http://your-s3-host.com:9000>

3. Complete Recommended
cn.conf
For the GC tool to work correctly with an HTTP-only, path-style S3 compatible storage, your file should look like this:

Copy code

properties
# S3 Connection
object_storage_endpoint = <host-ip-or-dns>:<port>
object_storage_access_key_id = <your-ak>
object_storage_secret_access_key = <your-sk>
object_storage_region = <your-region>

# Path Style & HTTP logic
aws_s3_use_virtual_hosted_style = false
aws_s3_use_https = false

# CN specifics (needed for the tool to initialize)
storage_root_path = /data/starrocks/storage
starlet_port = 9070

Why this fixes "Couldn't resolve host" When

aws_s3_use_virtual_hosted_style

true

(default), the client tries to resolve

dev-starrocks-fe.<your-endpoint>

. When it is set to

false

, the client resolves only

<your-endpoint>

, which matches the entry in your

/etc/hosts

or DNS, and then appends

/dev-starrocks-fe/

to the URL path of the request. References • deployment/shared_data/s3.md • deployment/shared_data/minio.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/BE_configuration.md • sql-reference/sql-functions/table-functions/files.md References • deployment/shared_data/s3.md • deployment/shared_data/minio.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/BE_configuration.md • sql-reference/sql-functions/table-functions/files.md References • deployment/shared_data/minio.md • administration/management/FE_configuration.md • deployment/shared_data/s3.md

Zurab Kiknavelidze

07/16/2026, 9:34 AM

@Rocky what should dry run via do_delete=false do? does not it list the potential delete files? it returns nothign

Rocky

07/16/2026, 9:34 AM

When running the

lake_datafile_gc

tool with

do_delete=false

, a "nothing" result usually indicates one of three things: either no garbage was found, the garbage is too new to be deleted, or you are looking at the wrong output file. Here is what you should expect and check: 1. Where to see the "List" The tool does not typically print the list of thousands of files directly to your terminal's standard output (stdout) to avoid flooding the console. * Check the Audit File: If you used the

--audit_file=./audit.txt

parameter, all potential delete candidates are written there. Check that file:

cat audit.txt

. * Check the Log: If you are running with

nohup

, check the

nohup.out

file. The tool will print a summary like:

Found X files, total size Y, to be deleted

. 2. The "Grace Period" (
expired_sec
) By default, the tool has an

expired_sec

parameter (often defaulting to 86400 seconds / 24 hours). * What it does: The tool will ignore any file that was created within the last 24 hours. This is a safety mechanism to prevent deleting data that is currently being written by an active compaction or load task. * Test: If you know you have old garbage, try setting

--expired_sec=0

for a dry run just to see if it catches anything. 3. The Recycle Bin (FE Metadata) StarRocks keeps metadata for dropped tables in a "recycle bin" before truly "forgetting" them. * The GC tool works by identifying files in S3 that are not referenced by any Tablet Meta. * If you dropped a table without using

FORCE

(e.g., just

DROP TABLE my_table

), the metadata remains in the FE recycle bin for a period (controlled by

catalog_trash_expire_second

, default is often 1–3 days). * As long as that metadata exists in the FE, the GC tool considers those files "active" and will not list them for deletion. 4. Verify the
root_path
Ensure your

root_path

matches the exact prefix where StarRocks stores data. * If your data is at

<s3://dev-starrocks-fe/starrocks_data/>

, but you run it on

<s3://dev-starrocks-fe/>

, the tool might not correctly map the structure. * Recommended: Run the tool on a specific table or partition path first to verify it's working:

--root_path="<s3://dev-starrocks-fe/1/10102/>"

(where

is DB ID and

is Table ID). Summary of what "Nothing" means: If the tool finishes with a message like

GC finished

but lists 0 files: 1. No Garbage: Your cluster is healthy and all files in S3 are currently referenced by active data or compaction tasks. 2. Too New: The files are garbage, but they were created less than 24 hours ago. 3. Still in Recycle Bin: The table was dropped recently, and the FE is still holding the "deletion" in its own trash. Next Step: Run the command again with

--expired_sec=0

and check the file specified in

--audit_file

. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • deployment/shared_data/s3.md References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/gcs.md • knowledge/admin_and_operation/shared_data_compaction.md

Zurab Kiknavelidze

07/16/2026, 9:35 AM

@Rocky I run this ./meta_tool.sh --operation=lake_datafile_gc --root_path="s3://prod-starrocks-int-sv/1c6aaddd-d45b-4520-9bcf-db33e64081b1/db3744741/3761620" --expired_sec=3600 --conf_file=../conf/cn.conf --do_delete=false and how to understand if it caught something?

Rocky

07/16/2026, 9:35 AM

To know if the

lake_datafile_gc

tool "caught something" (found garbage files) during a dry run (

do_delete=false

), you should look for specific indicators in the console output and logs. If the tool is returning "nothing," it typically means it either found 0 garbage files or there is a configuration/syntax issue preventing it from scanning. 1. What the "Success" Output Looks Like When the tool finishes, it should print a summary line to the console (stdout) or the log file. It looks something like this:

Copy code

text
GC finished. Found 150 useless files, total size: 10.5 GB.

* If you see `Found 0 useless files`: The tool successfully scanned the metadata and found that every file in that S3 path is currently "alive" (referenced by an active tablet) or is newer than your

--expired_sec=3600

(1 hour) limit. * If there is truly NO output: The tool may be failing silently or the output is being redirected to a log file. 2. Check the Audit File (Crucial) Your command is missing the

--audit_file

parameter. Without it, the tool doesn't have a designated place to write the specific list of files it intends to delete. Run it again with an audit file:

Copy code

bash
./meta_tool.sh --operation=lake_datafile_gc \
  --root_path="<s3://prod-starrocks-int-sv/1c6aaddd-d45b-4520-9bcf-db33e64081b1/db3744741/3761620>" \
  --expired_sec=3600 \
  --conf_file=../conf/cn.conf \
  --do_delete=false \
  --audit_file=./gc_scan_results.txt

After it finishes, check

gc_scan_results.txt

. If the file is empty, no garbage was found. 3. Possible Reasons for "Nothing Found" * Active Metadata: If the table/partition at that path is still active and hasn't had failed compactions or recent large deletes, there simply might not be any garbage. * The "Recycle Bin": If you recently dropped a table, its metadata stays in the FE's "recycle bin" (controlled by

catalog_trash_expire_second

) for a default of 1-3 days. As long as the FE remembers the table, the GC tool considers its files "valid" and will not catch them. * Path Syntax: In your command, you wrote

<s3://...>

(with angle brackets). If those brackets were literally in your command, the S3 client will fail to parse the path. Ensure the path is just

<s3://bucket/path>

. * Short Expiration:

--expired_sec=3600

is only 1 hour. StarRocks compaction often keeps historical versions for at least 30-60 minutes (

lake_autovacuum_grace_period_minutes

). If you want to be sure you catch everything, try

--expired_sec=0

during a dry run. 4. Where to check for errors If the command exits immediately with no output, check the BE/CN logs (usually in

../log/be.INFO

../log/be.out

relative to your tool location). Look for errors related to: *

S3 connection failed

Access Denied

Invalid S3 URI

Summary: If the summary says

Found 0 useless files

and the audit file is empty, your storage is clean according to the current FE's current metadata. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/Meta_recovery.md • administration/management/compaction.md

Zurab Kiknavelidze

07/16/2026, 9:45 AM

@Rocky I have run via do_delete-True but it kept still files

Rocky

07/16/2026, 9:45 AM

If the files still exist in S3 after running with

do_delete=true

, it is almost certainly because the tool did not consider those files to be garbage. The

lake_datafile_gc

tool is intentionally conservative to prevent data loss. Here are the specific reasons why it would "keep" files: 1. The FE Recycle Bin (Most Common) If you recently performed a

DROP TABLE

DROP DATABASE

without the

FORCE

keyword, the metadata is moved to the FE Recycle Bin. * The Logic: As long as the metadata exists in the recycle bin, the data files are still "referenced." The GC tool sees these files as valid and will not delete them. * The Fix: You must wait for the recycle bin to expire (default is 24 hours, controlled by

catalog_trash_expire_second

fe.conf

) or have used

DROP TABLE ... FORCE

. * How to check: Run

SHOW BACKENDS;

and check if there is a

Trash

size, or check

information_schema.recyclebin_catalogs

to see if your table is listed there. 2. Files are still "Referenced" by Tablet Meta The tool works by downloading the

.meta

files (Tablet Meta) from S3 and comparing them against the

.dat

files (Data files). * If a

.dat

file is mentioned in any active Tablet Meta, the tool will not delete it, even if you think it's old. * This happens if a compaction task failed but the original files are still part of the "visible version" of the table. 3. The
expired_sec
Safety Window You set

--expired_sec=3600

(1 hour). * The tool ignores any file with a "Last Modified" timestamp in S3 that is newer than 1 hour ago. * If you just performed a large operation (like a failed load or a drop) 30 minutes ago, those files are protected by this window. 4. Compaction Grace Period StarRocks has a setting called

lake_autovacuum_grace_period_minutes

(default is usually 5–30 mins). * During this window, even after a successful compaction, the system keeps old versions of files to ensure that long-running queries don't fail with "meta does not exist." * If the GC tool runs while these versions are still within their grace period, it may skip them. How to Verify What Happened 1. Check the Audit File: Re-run the tool with

--audit_file=./gc_report.txt

. * Open

gc_report.txt

. If the file is empty, the tool did not find any garbage. * If the tool found files but failed to delete them (due to permissions), it will log errors in the BE/CN logs (

<http://be.INFO|be.INFO>

). 2. Verify S3 Permissions: Ensure the credentials in your

cn.conf

have

s3:DeleteObject

permissions. If the tool "tries" to delete but fails, it will show errors in the console/log. 3. Identify the files: Pick one file that you think should have been deleted and check its "Last Modified" time in S3. If it's newer than your

expired_sec

, that's your answer. Recommendation: If you are sure you want to clear everything from a specific table that you already dropped, check the recycle bin:

Copy code

sql
-- Check if the table is still in the trash
SELECT * FROM information_schema.recyclebin_catalogs;

If it is there, you can either wait 24 hours or (if you haven't dropped it yet) use

DROP TABLE <name> FORCE

. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • sql-reference/information_schema/recyclebin_catalogs.md • administration/Data_recovery.md • data_source/catalog/iceberg/iceberg_meta_table.md • faq/operation_maintenance_faq.md

Rocky

07/16/2026, 9:45 AM

• knowledge/admin_and_operation/shared_data_compaction.md

7 Views

Open in Slack

Previous Next