@Rocky auditloader data starrocks_audit_db__.starrocks_audit_tbl__ is saved in object store in case of shared architechture right?

Yes, that is correct. In a StarRocks shared-data (compute-storage decoupled) architecture, the data for the audit table (

starrocks_audit_db*.starrocks_audit_tbl*

) is saved in the object store (such as S3, GCS, Azure Blob Storage, or MinIO). Here is how the process works in a shared-data environment: 1. Local Generation: Each Frontend (FE) node initially generates audit logs as local text files located at

fe/log/fe.audit.log

. 2. Plugin Processing: The AuditLoader plugin runs as a background thread on the FE. it reads these local logs and loads them into the StarRocks cluster using the

Stream Load

mechanism. 3. Persistent Storage: Since the cluster is in shared-data mode, any table created (including the audit table) is a "cloud-native" table. This means the data is persistently stored in the configured object store, while local disks on the Compute Nodes (CNs) are only used for caching hot data to accelerate queries. Verification You can verify the storage path of your audit table by running the following command in your MySQL client:

sql
SHOW PROC '/dbs/starrocks_audit_db__';

Look for the

StoragePath

column for the

starrocks_audit_tbl__

row. It will show the URI (e.g., `s3://your-bucket/...`) where the audit data is actually stored. Important Configuration Note When creating the audit table in a shared-data cluster, ensure you set

"replication_num" = "1"

. In shared-data mode, the object store handles data reliability, so the multiple-replica mechanism used in shared-nothing (Local BE storage) is not required. References • introduction/Architecture.md • table_design/table_design.md • deployment/shared_data/s3.md • faq/shared_data_faq.md • quick_start/shared-data.md • deployment/shared_data/minio.md

@Rocky how it comes that I can run

ALTER TABLE starrocks_audit_db__.starrocks_audit_tbl__
DROP PARTITION p20260702

and it works without connecting to s3?

In a StarRocks shared-data architecture, the

ALTER TABLE ... DROP PARTITION

command works without you needing a direct S3 connection because it is primarily a metadata operation handled by the StarRocks cluster itself. Here is the breakdown of why this works: 1. Metadata vs. Physical Data When you run the

DROP PARTITION

command: * Metadata Change: The Frontend (FE) node immediately updates its internal metadata catalog to mark that partition as "deleted" or "dropped." This happens in the FE's memory and its local BDBJE metadata store. * Logical Deletion: Once the metadata is updated, the partition becomes invisible to all subsequent queries. The command returns "OK" because this logical step is finished. 2. The Cluster Already Has the Credentials Your SQL client (like MySQL CLI or DBeaver) only talks to the StarRocks Frontend (FE). * The FE (and the Compute Nodes) are already configured with S3/Object Store credentials via the

STORAGE VOLUME

or cluster configuration files (`fe.conf`/`be.conf`). * The cluster has the "connection" to S3; you as the user only need the "connection" to the cluster. 3. Asynchronous Physical Deletion (Vacuum) The physical data in the object store is not usually deleted the instant you run the command. * Background Cleanup: StarRocks uses a background process called Vacuum (or Garbage Collection). * Retention Period: Dropped partitions/tables are often moved to a metadata "recycle bin" for a period (controlled by

catalog_trash_expire_second

in

fe.conf

, default is 24 hours). * Final Deletion: Only after this expiration period does a background thread in the cluster issue the actual

DeleteObject

calls to the S3 API to remove the files. Summary You don't need an S3 connection because: 1. You are only telling the FE to update its "map" of the data. 2. The FE/CN nodes already have the credentials to talk to S3 for you. 3. The actual heavy lifting of deleting files from S3 happens asynchronously in the background. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/feature-support-shared-data.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/resource_isolation.md • administration/management/resource_management/workload_group.md • administration/management/resource_management/resource_isolation_CN.md • administration/management/resource_management/resource_isolation_BE.md • administration/management/resource_management/resource_isolation_FE.md • administration/management/resource_management/resource_isolation_manual.md • administration/management/resource_management/resource_isolation_SQL.md • administration/management/resource_management/resource_isolation_REST_API.md

@Rocky so what is saved in object store and what internally in case of shared architechture meta?

In a StarRocks shared-data (compute-storage decoupled) architecture, the "meta" and "data" are distributed between your local FE disks and the object store as follows: 1. Saved in the Object Store (Shared) This is the persistent, "source of truth" storage for your table content. * Table Data: The actual rows of your audit logs (and all other tables) are stored as segment files in the object store. * Tablet Metadata (Lake Metadata): Since v3.1.4, each partition directory in the object store contains a

meta/

sub-directory. This stores the specific metadata for those tablets (like which segment files belong to which version). * Recycle Bin: When you drop a partition, the reference is removed from the FE, but the actual files stay in the object store's "trash" for a period (e.g., 24 hours) before being physically deleted. 2. Saved Internally (Local FE/CN Disks) These are used for cluster management, query planning, and performance acceleration. * Global Metadata (BDBJE): The Frontend (FE) nodes store the "Catalog" metadata in their local

meta/

directory (using Berkeley DB Java Edition). This includes: * Database and Table definitions (schemas). * User accounts and permissions. * Resource group configurations. * Transaction logs (Edit Logs). * Data Cache: The Compute Nodes (CNs) use their local disks to store a cache of the segment files downloaded from the object store. When you query the audit table, the CNs check this local cache first. * FE Memory: A complete copy of the Global Metadata is kept in the FE's RAM to ensure extremely fast query planning. Summary Comparison Component | Shared (Object Store) | Internal (Local FE/CN Disks) Audit Table Rows | Yes (Segment files) | No (Only as temporary cache) Table Schema | No | Yes (FE BDBJE / RAM) Tablet/File Mapping | Yes (Lake Tablet Meta) | No User/Permissions | No | Yes (FE BDBJE) Hot Data Cache | No | Yes (CN Disk Cache) Why your

DROP PARTITION

worked without "connecting" to S3: When you ran that command, you only updated the FE's Internal Metadata (the schema and partition list). The FE simply "unlinked" that partition in its local database. It didn't need to talk to S3 at that exact moment to make the data disappear from your queries. The cluster's background "vacuum" process will use the cluster's stored S3 credentials to delete the physical files later. References • introduction/Architecture.md • faq/shared_data_faq.md • quick_start/shared-data.md • table_design/table_design.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md References • faq/shared_data_faq.md • quick_start/shared-data.md • introduction/Architecture.md • table_design/table_design.md • deployment/shared_data/s3.md

@Rocky what about views? I cant select view because [42000][1064] starlet err [RequestID=18BEC82FB267F07EAA482352][StatusCode=404]Get object s3://dev-starrocks-fe/meta/fe919e73-9b67-4920-b15b-fad76a32c664/db866522/868927/2772109/meta/0000000000000000_0000000000000002.meta error: The specified key does not exist.: BE:10001

The error you are seeing (

StatusCode=404 Get object ... .meta ... The specified key does not exist

) is a classic sign of a metadata desynchronization between the StarRocks Frontend (FE) and the Object Store in a shared-data cluster. Even though you are selecting from a View, the error is actually occurring when StarRocks tries to access the underlying table data that the view points to. Why this happens (The "Vacuum" Race Condition) In StarRocks shared-data architecture, data is versioned. When new data is loaded (like by the AuditLoader) or when background compaction runs, new versions of the data and metadata are created in S3. The old versions are eventually deleted by a background process called Autovacuum. If a query (like your

SELECT

on the view) is still running or starts just as the Autovacuum process deletes the metadata file (

.meta

) for an older version that the FE still thinks is valid, you get a 404 error. This is especially common with the AuditLoader table because it writes data very frequently, leading to a high number of versions and aggressive compaction/vacuuming. How to Fix It You need to give queries more "breathing room" by delaying how quickly StarRocks deletes old metadata files from S3. 1. Increase the Vacuum Grace Period: Modify the following configuration in your

fe.conf

(or via

ADMIN SET FRONTEND CONFIG

) to keep old metadata files longer:

sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");

The default is often 5 minutes. Increasing this to 60 minutes ensures that even if a version is compacted, its files stay in S3 for an hour, allowing queries to finish safely. 2. Check for Clock Skew: Ensure that the clocks on all FE nodes and the object storage (S3) are synchronized (using NTP). If an FE node's clock is ahead, it might trigger the vacuum process earlier than it should. 3. Audit Table Specifics: If your view is querying the

starrocks_audit_tbl__

, remember that this table is modified constantly. If the query in your view is very complex or slow, the underlying data "shifts" underneath it. Increasing the grace period above is the most effective fix for this. Summary of Storage for Views * The View Definition: Saved internally (FE local disks/RAM). It is just a SQL string. * The Error: Is not about the view itself being missing from S3. It is about the table data/metadata referenced by the view having been deleted from S3 before the query could read it. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md • table_design/table_design.md • administration/Meta_recovery.md • quick_start/shared-data.md • introduction/Architecture.md • deployment/shared_data/s3.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/resource_isolation.md