@Rocky does starrocks latest version support in shared mode when data is in s3 a TTL lets say 7 days that after that it deleted from s3, does it support keeping data in disk forever also if we are in shared mode (tables are in s3 thats fine but we want to get the data really fast we want it to almost always be on disk -> non shared mode has issues of scalling up and down and its not automatic no ? so thats not good), does it support multiple paritions for the same table , so we cant do drop by parittion of recevie_ts but query by parittions of evenstsrttime for late ariving data ? (another job would would once a day find partitions where recevie_ts is older than 7 days, we wont do drop partition by evenstsrttime since data can arrive late (sone days later))

StarRocks latest versions (v3.x) fully support your requirements for shared-data mode (Compute-Storage Separation), specifically regarding S3 lifecycle management, local disk caching, and multi-column partitioning. 1. TTL and S3 Data Deletion StarRocks supports automatic data lifecycle management in shared-data mode: * Automatic Dropping: You can use Expression Partitioning with the

partition_live_number

property. For example, setting it to

7

will keep only the 7 most recent partitions. * Manual/Scripted Dropping: You can use

ALTER TABLE ... DROP PARTITION

. * S3 Cleanup: In shared-data mode, when a partition is dropped, StarRocks does not just delete metadata. A background process called Vacuum eventually deletes the orphaned data files from S3. To delete data immediately, you can use the

FORCE

keyword (though usually applied to tables). 2. Fast Access: Keeping Data on Disk "Forever" Shared-data mode uses a Data Cache (on local BE/CN disks) to provide performance similar to local storage while maintaining the scaling benefits of S3. * Cache Persistence: Set

datacache.enable = "true"

(default). To keep data in the cache as long as possible, you can set

datacache.partition_duration

to a very long time (e.g.,

100 YEAR

) or leave it empty so all data is treated as "hot." * Behavior: The data is stored in S3 (source of truth) but cached on local disks. If you scale up/down by adding or removing CN (Compute) nodes, the new nodes will pull data from S3 into their local cache as queries run. * Warmup: In StarRocks v3.3+, you can use the

CACHE SELECT

command to proactively "warm up" the local disk cache so that queries are fast immediately after scaling. 3. Multiple Partitions (Multi-column Partitioning) You can achieve your goal of querying by

event_time

while dropping by

receive_ts

using Multi-column Expression Partitioning (supported since v3.1, with expanded support in v3.4+). How to set it up: You can partition by both columns. From v3.4 onwards, StarRocks supports mixed expressions where one column can be a time function.

sql
CREATE TABLE my_table (
    events_start_time DATETIME,
    receive_ts DATETIME,
    ...
) 
PARTITION BY 
    date_trunc('day', events_start_time), 
    date_trunc('day', receive_ts)
DISTRIBUTED BY HASH(events_start_time) BUCKETS 10
PROPERTIES (
    "datacache.enable" = "true"
);

Handling your scenario: 1. Querying: When you query by

events_start_time

, StarRocks will use partition pruning to only look at the relevant partitions. 2. Dropping Late Data: Because the partition name includes both values (e.g.,

p20231001_20231005

), you can run a maintenance job (once a day) that identifies and drops partitions where the

receive_ts

part is older than 7 days. * Note: Since

receive_ts

is part of the partition key, late-arriving data for an old

event_time

will simply go into a new partition with a current

receive_ts

, which keeps it safe from your 7-day cleanup logic until 7 days after it actually arrived. Summary of Benefits for Your Case * Scaling: Use CN nodes for automatic/easy scaling. Since data is in S3, you don't have to worry about the heavy data redistribution (rebalancing) that occurs in non-shared (shared-nothing) mode. * Performance: The local Data Cache ensures that for your active 7-day window, the data is served directly from the local NVMe/SSD disks of your CN nodes. * Flexibility: Multi-column partitioning allows you to decouple the "Business Time" (for queries) from the "System Time" (for TTL/Cleanup). References • table_design/data_distribution/dynamic_partitioning.md • table_design/data_distribution/expression_partitioning.md

@Rocky when is it right to use non shared mode, and when to use non shared. we got data in starrocks and want to drop old and move it to iceberg - thats our use case.

For your use case—where you want fast performance (on-disk) for recent data and the ability to archive old data to Iceberg—*Shared-data mode* is almost certainly the right choice, but here is the breakdown of when to choose each. 1. When to use Shared-Data vs. Shared-Nothing Feature | ***Shared-Data** (Compute-Storage Separation)* | ***Shared-Nothing** (Classic/Non-shared)* Storage Source | Object Storage (S3/GCS/Azure) | Local Disks on BE nodes Performance | Very High (when data is in local cache) | High/Consistent (data is always local) Scaling | Instant. Add CN nodes in seconds. No data rebalancing. | Slow. Adding BEs requires migrating data across the network. Storage Cost | Low. S3 is much cheaper than local NVMe/SSD. | High. Limited by local disk capacity on BE nodes. Best For | Cloud-native, elastic workloads, large datasets where only a subset is "hot." | On-prem without S3, stable workloads that rarely scale, or ultra-low latency requirements. For your case: Since you want "almost always on disk" but also want automatic scaling, Shared-Data mode with Data Cache is the winner. You get the speed of local disk (via cache) and the elasticity of the cloud (via CN nodes). ────────── 2. Archiving Data to Iceberg (Your Use Case) Since you want to move data to Iceberg and then drop it from StarRocks, you can follow this workflow using StarRocks as the engine: Step 1: Create an Iceberg Catalog First, point StarRocks to your Iceberg warehouse (e.g., using Glue, Hive Metastore, or Tabular).

sql
CREATE EXTERNAL CATALOG iceberg_catalog
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "glue", -- or hive, rest, etc.
    "aws.s3.region" = "us-east-1"
);

Step 2: Move the Data (Insert into Iceberg) StarRocks supports

INSERT

into Iceberg tables (v3.1+). You can run a daily job to archive data older than 7 days:

sql
INSERT INTO iceberg_catalog.iceberg_db.archived_table
SELECT * 
FROM starrocks_db.fast_table
WHERE receive_ts < CURRENT_DATE() - INTERVAL 7 DAY;

Note: This command reads from StarRocks and writes directly to Iceberg in Parquet format. Step 3: Drop the Old Data from StarRocks After the move is successful, drop the partitions from StarRocks:

sql
ALTER TABLE starrocks_db.fast_table 
DROP PARTITION p20231001; 
-- Or use the automated TTL 'partition_live_number' as mentioned before.

Why this fits you: 1. Fast Queries: While data is in StarRocks, it is served from the local NVMe cache of your CN nodes. 2. Scaling: If you have a spike in queries, you can spin up 10 more CN nodes instantly. They will pull data from S3 into their local caches and start helping immediately. 3. Archival: StarRocks acts as the "bridge." It can read its own high-performance format and write to the open Iceberg format seamlessly. 4. No Scaling Pain: In shared-nothing mode, if you added nodes to handle a query spike, you'd have to wait hours for the 7 days of data to "rebalance" to the new nodes. In shared-data, there is zero rebalancing. References • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/s3.md • quick_start/shared-data.md