@Rocky why dont have scheduled delete in starrocks ?

Actually, StarRocks does have built-in mechanisms for scheduled data deletion, but they are designed around the performance characteristics of an OLAP engine. Instead of a generic

DELETE

scheduler for individual rows, StarRocks primarily uses Dynamic Partitioning and TTL (Time-to-Live) management. 1. The Recommended Way: Dynamic Partitioning For time-series or high-volume data, you should use Dynamic Partitioning. This feature automatically creates new partitions and drops expired partitions based on a defined schedule and retention period. * Why use this? Dropping a partition is a metadata operation that is nearly instantaneous and has zero overhead. In contrast, running a

DELETE

command on millions of rows is very expensive because it creates "tombstones" (delete markers) and requires heavy background compaction to physically remove the data. * Example configuration:

sql
    PROPERTIES (
        "dynamic_partition.enable" = "true",
        "dynamic_partition.time_unit" = "DAY",
        "dynamic_partition.end" = "3",      -- Create partitions for the next 3 days
        "dynamic_partition.history_number" = "30" -- Keep only the last 30 days of data
    );

Any data older than 30 days is automatically deleted. 2. The New "Task" Framework (v3.3+) StarRocks recently introduced a built-in Task/Job framework (

SUBMIT TASK

). While it is currently optimized for ETL (like

INSERT INTO ... SELECT

), it provides the foundation for native SQL scheduling. * For now, if you need to run a complex

DELETE

or

TRUNCATE

periodically, most users use external orchestrators like Airflow, DolphinScheduler, or simple Cron jobs to send the SQL command to StarRocks. 3. Why isn't row-level scheduled

DELETE

the default? There are architectural reasons why StarRocks discourages frequent, automated row-level deletes: * Compaction Overhead: StarRocks is a columnar storage engine. When you delete rows, they aren't removed immediately; they are marked as deleted. The system must perform a "compaction" (rewriting data files) to physically remove them, which consumes significant CPU and I/O. * Version Bloat: Frequent

DELETE

operations create many data versions, which can slow down query performance until compaction catches up. * Best Practice: The official recommendation is: "Do not use

DELETE

to clean up data; use

TRUNCATE PARTITION

instead." Summary of Options Requirement | Solution Delete by age (e.g., keep 30 days) | Use Dynamic Partitioning (built-in, high performance). Delete specific rows on a schedule | Use an External Scheduler (Airflow/Cron) to call

DELETE

during off-peak hours. Manage storage cleanup | Tune FE/BE parameters like

catalog_trash_expire_second

to control how long "dropped" tables stay in the recycle bin. For more details on data lifecycle management, you can refer to the Dynamic Partitioning documentation. References • sql-reference/information_schema/fe_tablet_schedules.md • sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md • sql-reference/sql-statements/loading_unloading/ETL/ALTER_TASK.md • sql-reference/information_schema/tasks.md