@Rocky can we run the validation sql task in starocks only by its own scehduling?

Yes, you can run and schedule validation SQL tasks using StarRocks' built-in scheduler. Starting from v3.3, StarRocks supports periodic scheduling for asynchronous tasks using the

SUBMIT TASK

statement. This allows you to run SQL logic at regular intervals (e.g., every minute, hour, or day) without needing an external orchestrator like Airflow or a cron job. How to Run a Validation Task Since

SUBMIT TASK

currently supports ETL statements (

INSERT

or

CREATE TABLE AS SELECT

) and

CACHE SELECT

, you cannot run a raw

SELECT

statement that only returns a result set to the console. For validation purposes, the standard pattern is to use an

INSERT INTO ... SELECT

statement to write the results of your validation check (e.g., counts of nulls, duplicates, or out-of-range values) into a dedicated validation or logging table. Example: Scheduling a Data Quality Check If you want to check for data anomalies every hour and store them in a log table:

sql
SUBMIT TASK hourly_data_validation
SCHEDULE EVERY (INTERVAL 1 HOUR)
AS 
INSERT INTO validation_log (check_time, table_name, error_count)
SELECT 
    NOW(), 
    'sales_orders', 
    COUNT(*) 
FROM sales_orders 
WHERE order_amount < 0 OR customer_id IS NULL;

Key Features of the Built-in Scheduler * Asynchronous Execution: Tasks run in the background and do not block your current session. * Flexible Scheduling: You can specify a start time with

START('YYYY-MM-DD HH:MM:SS')

and an interval with

EVERY(INTERVAL ...)

. * Monitoring: You can track the status and history of your validation tasks using system views: _

SELECT _ FROM information_schema.tasks;

(View task metadata) _

SELECT _ FROM information_schema.task_runs;

(Check execution status: SUCCESS, FAILED, RUNNING) Considerations 1. Supported Versions: * v2.5+: Supports manual asynchronous task submission. * v3.3+: Supports periodic scheduling. 2. Statement Limits: Currently, the scheduler only supports

INSERT

,

CREATE TABLE AS SELECT

(CTAS), and

CACHE SELECT

. It does not support generic

SELECT

statements for display or

UPDATE/DELETE

statements. 3. Permissions: You need the

TASK

privilege to create and manage tasks. 4. Error Handling: If the SQL logic within the task fails (e.g., a schema mismatch during insertion), the task run will be marked as

FAILED

in

information_schema.task_runs

. If your validation requirement involves complex alerting (e.g., sending an email or Slack message when a count > 0), you would still need an external tool to monitor the

validation_log

table or the

task_runs

view, as StarRocks does not currently have a built-in alerting/notification system. References • knowledge/admin_and_operation/shared_data_compaction.md