@Rocky when I create TEMPORARY tables in custom database tmp in shared architechture it creates this sessioned tables in that storage volume under tmp database right? and after session is finished I see that there are SCHEMA files left in the object store and why vacuum does not clean it up?

In StarRocks shared-data architecture (Compute-Storage Separation), temporary tables follow the same physical storage principles as standard "Lake" tables, which leads to the behavior you've observed. 1. Why are "SCHEMA files" left in the Object Store? When you create a temporary table in a shared-data cluster, StarRocks stores its data and metadata in the object storage (e.g., S3) under a path corresponding to the database and table ID (e.g.,

s3://.../db_id/table_id/...

). The "SCHEMA files" you see are likely the tablet metadata files (typically named

{tablet_id}_{version}.meta

) stored in the

meta/

subdirectory of the table path. The lifecycle behavior is as follows: * Session End: When your session ends, StarRocks automatically triggers a

DROP TABLE

for the temporary table. * Recycle Bin: By default, a

DROP TABLE

operation in StarRocks does not immediately delete physical data. Instead, it moves the table metadata to the Catalog Recycle Bin (Trash). * Retention Period: The metadata remains in the recycle bin for a period defined by the FE configuration

catalog_trash_expire_second

(the default is 86,400 seconds or 24 hours). Because the metadata still exists in the recycle bin, the table is technically still "referenced" by the system's metadata management. 2. Why doesn't VACUUM clean them up? In shared-data architecture, the

VACUUM

process (or

lake_autovacuum

) is designed to clean up unreferenced files—files that are not linked to any active version or metadata. * As long as the temporary table's metadata exists in the Recycle Bin, the files (including the

.meta

schema files and data segments) are considered "referenced." *

VACUUM

will only purge these files from the object store after the metadata has expired and been removed from the recycle bin. How to manage this If you want these files to be cleaned up more aggressively or immediately, you have a few options: 1. Manual Force Drop: Instead of waiting for the session to close, manually drop the table using the

FORCE

keyword:

sql
    DROP TEMPORARY TABLE IF EXISTS tmp.your_table_name FORCE;

The

FORCE

keyword bypasses the recycle bin and triggers immediate deletion of both metadata and data. 2. Adjust Recycle Bin TTL: You can shorten the time metadata stays in the trash by adjusting the FE configuration. This will affect all dropped tables in the cluster:

sql
    -- Example: Set to 1 hour (3600 seconds)
    ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "3600");

3. Check Vacuum Grace Period: Ensure the vacuum process itself isn't waiting too long. The FE configuration

lake_autovacuum_grace_period_minutes

(default is 5 minutes) controls how long

VACUUM

waits after a file becomes unreferenced before deleting it. Summary: The files are left behind because the automatic cleanup at session end currently uses a standard "soft" drop, keeping the table in the FE's recycle bin for 24 hours.

VACUUM

respects this safety window to allow for potential metadata recovery. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky no I'm talking about SCHEMA_ files not in meta

In StarRocks shared-data (lake) architecture, the behavior you're seeing—where files with a

SCHEMA_

prefix remain in the object store after a temporary table's session has ended—is related to the Tablet Schema Reuse feature. 1. What are these

SCHEMA_

files? In shared-data mode (starting around version 3.1+), StarRocks optimizes storage by separating the Tablet Schema from the Tablet Metadata (

.meta

files). * Tablet Metadata (

meta/

): Contains versioning, rowset, and segment information specific to a single tablet. * Tablet Schema (

SCHEMA_

): Contains the actual column definitions and types. Because many tablets (across different partitions or even different tables) often share the exact same schema, StarRocks saves the schema as a standalone object in the object store using a content-addressed naming convention (e.g.,

SCHEMA_<checksum>

). This avoids duplicating the same schema data thousands of times in every

.meta

file. These files are typically stored at the Database level or Table level in the object store. 2. Why does the session end not delete them? When a session ends, the temporary table is "dropped." As mentioned in the previous response, this typically moves the metadata to the Catalog Recycle Bin (FE side) for 24 hours (default:

catalog_trash_expire_second

). However, even after the table metadata is physically deleted from the recycle bin, the

SCHEMA_

files may persist because: * Shared Lifecycle: Because these files are "reusable," the system doesn't necessarily know if only your temporary table was using that specific schema. * Reference Counting: The garbage collection (vacuum) for shared schema files often works on a different cadence or has a stricter "safety period" than data segments. If the system thinks the schema might be needed for a pending transaction or a different tablet with the same structure, it will not delete it immediately. 3. Why doesn't VACUUM clean them up? The

lake_autovacuum

process (which handles cleaning up unreferenced files in S3/object storage) primarily focuses on data segments and tablet metadata. * Grace Period: Vacuuming in shared-data mode respects a grace period (configured via

lake_autovacuum_grace_period_minutes

, default is usually 5 minutes, but can be much longer depending on cluster settings). * Metadata Persistence: As long as the dropped table is still in the FE's recycle bin (the 24-hour window), the vacuum process considers all associated files "referenced." * Schema Registry: The shared schema files are managed by an internal registry. In some earlier 3.x versions, the cleanup logic for these specifically was less aggressive than for data segments to prevent accidental data loss if a schema was reused. How to address this: If you want to ensure the entire footprint of the temporary table is removed: 1. Use `DROP TEMPORARY TABLE ... FORCE`: If you manually drop the table with

FORCE

before the session ends, it instructs the FE to skip the recycle bin and notify the storage layer to delete all associated metadata immediately. 2. Shorten Recycle Bin TTL: Reduce

catalog_trash_expire_second

in FE configuration if you do not need the 24-hour safety window for recovery. 3. Check `lake_tablet_schema_reuse_enable`: If you don't want this feature at all (though it is recommended for performance), you can check if this BE/FE parameter is enabled. Disabling it would bundle the schema back into the

.meta

files, but this is generally not recommended for large-scale clusters. Summary: You are seeing the physical manifestation of "Schema Reuse." The

SCHEMA_

files are left behind because the table is likely still in the FE's recycle bin, or the vacuum's safety grace period has not yet elapsed for those shared objects. References • table_design/table_types/primary_key_table.md

@Rocky so whats the path in storage volumes? db/table/tablet/meta?

Yes, you are correct in your observation. In the StarRocks shared-data (Lake) architecture, the storage layout and the lifecycle of "Schema" files differ slightly from the data segments. 1. The Path in Storage Volumes Since StarRocks v3.1.8 and v3.2.3, the storage path for a table is organized hierarchically to prevent a single directory from becoming too large. The typical structure is:

s3://[bucket_path]/[cluster_id]/db[db_id]/[table_id]/[partition_id]/

Inside this path, you will find: *

data/

: Contains the physical data segments (e.g.,

.dat

files). *

meta/

: Contains tablet metadata files (

{tablet_id}_{version}.meta

). Where are the

SCHEMA_

files? Because StarRocks uses Tablet Schema Reuse, the schema (column names, types, etc.) is stripped out of the individual

.meta

files to save space and improve performance. These files (prefixed with

SCHEMA_

) are stored at a higher level—typically at the Database root or *Table root*—so they can be shared by all tablets in that table (or even across the database). The path usually looks like: *

s3://[bucket_path]/[cluster_id]/db[db_id]/SCHEMA_[checksum]

* OR

s3://[bucket_path]/[cluster_id]/db[db_id]/[table_id]/SCHEMA_[checksum]

2. Why doesn't Vacuum clean them up? Even if your temporary table session has ended and the table is "dropped," the

SCHEMA_

files may persist for two main reasons: * The Catalog Recycle Bin (Trash): When a temporary table is dropped, StarRocks does not immediately delete its metadata. It moves it to the internal Recycle Bin for a safety period (default 24 hours, controlled by

catalog_trash_expire_second

). As long as that metadata exists in the trash, the

VACUUM

process (lake autovacuum) considers the associated

SCHEMA_

and data files "referenced" and will not delete them. * Shared Reference: Because these files are "reusable," the system's garbage collection for schemas is often more conservative. If another active table or partition shares the exact same schema structure, that

SCHEMA_

file will remain indefinitely because it is still being used by other live objects. 3. How to Force Cleanup If you want to ensure these files are removed immediately after your temporary session: 1. Force Drop: Before the session ends, manually run:

sql
    DROP TEMPORARY TABLE tmp.your_table_name FORCE;

The

FORCE

keyword bypasses the recycle bin, allowing

VACUUM

to identify the files as unreferenced much sooner. 2. Adjust Trash TTL: You can reduce the global metadata retention time (not recommended for production, but useful for test environments):

sql
    ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "60"); -- 1 minute

Summary: You see them at the database/table level because they are shared metadata. They aren't cleaned up immediately because the table is likely still "alive" in the FE's recycle bin for the default 24-hour window. References • faq/shared_data_faq.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/s3.md • deployment/shared_data/azure.md • deployment/shared_data/hdfs.md • deployment/shared_data/gcs.md • administration/management/FE_configuration.md

@Rocky when I run drop table FORCE after what time it cleans up object storage?

When you run

DROP TABLE ... FORCE

(or

DROP TEMPORARY TABLE ... FORCE

), the cleanup process in a StarRocks shared-data cluster moves through two stages. While the metadata is removed instantly, the physical files in your object storage (S3/Azure/GCS) are deleted asynchronously based on the following background cycles: 1. Metadata Removal (Instant) Unlike a normal

DROP

, which keeps the table in the Catalog Recycle Bin for 24 hours (default

catalog_trash_expire_second

), the

FORCE

command deletes the table metadata from the FE (Frontend) immediately. From this moment on, the table is "invisible" and cannot be recovered. 2. Object Storage Cleanup (30+ Minutes) Even after the metadata is gone, the physical data segments and

SCHEMA_

files remain in the object store until the Lake AutoVacuum process identifies them as "unreferenced." The timing is governed by these configurations: *

lake_autovacuum_grace_period_minutes

(Default: 30 minutes): This is the most critical delay. Even if a file is unreferenced, StarRocks waits for this grace period before physically deleting it to ensure that long-running queries or late-arriving tasks that might still have a handle on those files do not fail. _ _Note:* In versions earlier than v3.1.10/v3.2.5, this was 5 minutes, but it has been increased to 30 minutes for safety. *

lake_autovacuum_partition_naptime_seconds

(Default: 180 seconds): This is the interval at which the background vacuum thread wakes up to check for garbage. Why you still see

SCHEMA_

files The

SCHEMA_

files you mentioned are part of the "Tablet Schema Reuse" feature. Because these are stored at the Database or Table level rather than inside the tablet folder: 1. Safety Buffer: The vacuum logic for shared schemas is often even more conservative than for data segments. 2. Shared References: If other tables in the same database share the same schema structure (common with temporary tables generated by the same script), the

SCHEMA_

file will not be deleted because it is still "referenced" by other live metadata. 3. Vacuum Cycle: If the

lake_autovacuum_grace_period_minutes

has not passed, the vacuum thread will simply skip these files during its current run. Summary of Timing Action | Metadata Removal | Object Storage (S3) Deletion Standard DROP | Instant (moves to Trash) | ~24 hours + 30 mins DROP ... FORCE | Instant | ~30 to 45 minutes Recommendation: If you need the space cleared faster in a dev/test environment, you can temporarily reduce the FE configuration

lake_autovacuum_grace_period_minutes

to a lower value (e.g., 1 or 5), but be aware this may cause active queries on other tables to fail if they are running during a vacuum cycle. longer than that. . running during a compaction is is. long-running. References • faq/shared_data_faq.md • faq/Others.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky

lake_autovacuum_partition_naptime_seconds

thats 180

The

lake_autovacuum_partition_naptime_seconds

(default 180 seconds) is indeed the heartbeat for the vacuum process, but it works in tandem with the grace period to determine when those

SCHEMA_

files actually disappear from S3/object storage. Here is exactly how the timing works when you run `DROP TABLE ... FORCE`: 1. The 180-second "Nap" (Check Interval) The

lake_autovacuum_partition_naptime_seconds

controls how often the FE (Leader) wakes up to scan for garbage. * Because your

SCHEMA_

files are stored at the Table or Database level, they aren't tied to a specific partition's "nap." * Instead, when the table is dropped, the metadata is removed. The next time the global vacuum cycle runs (which is also influenced by this interval), it identifies the

SCHEMA_

file as a candidate for deletion. 2. The 30-minute "Grace Period" (Wait Time) Even if the vacuum thread wakes up every 180 seconds, it will not delete the

SCHEMA_

file immediately. It honors the

lake_autovacuum_grace_period_minutes

(Default: 30 minutes). * Why? If a query was starting exactly at the moment you ran

DROP TABLE FORCE

, it might still need to read that schema or data file. StarRocks keeps "unreferenced" files for 30 minutes to prevent "file not found" errors for edge-case queries. 3. Total Time to Cleanup After running

DROP TABLE ... FORCE

, you can expect the files to be physically deleted from object storage in:

lake_autovacuum_grace_period_minutes

+

lake_autovacuum_partition_naptime_seconds

*(Typically 33 minutes with default settings).* Why

SCHEMA_

files might stick around even longer: If you still see them after 45–60 minutes, check these two things: 1. Schema Reusability: If you have any other table (temporary or permanent) in that same database with the exact same schema (same columns, same types, same order), they are sharing that

SCHEMA_<checksum>

file. As long as at least one table exists that uses that schema, the file will never be deleted. 2. Catalog Recycle Bin: If you forgot the

FORCE

keyword, the table is in the "trash" for 24 hours. The

SCHEMA_

file is considered "in use" until the table is physically purged from the trash. How to speed it up (for testing) If you are in a dev environment and want them gone now, you can temporarily change the FE configuration:

sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1");

Note: After changing this, wait about 5 minutes (the naptime + the new grace period) and the files should be cleaned up. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md • faq/shared_data_faq.md References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md