Lý Thành
07/29/2026, 3:16 AMRocky
07/29/2026, 3:16 AMLý Thành
07/29/2026, 3:16 AMRocky
07/29/2026, 3:16 AMenable_vector_index is not a valid system variable in StarRocks.
Starting from v3.4, vector indexing is supported as a beta feature. To enable it, you must use the FE configuration item enable_experimental_vector.
How to enable vector indexing
You can enable it dynamically using the following command:
sql
ADMIN SET FRONTEND CONFIG ("enable_experimental_vector" = "true");
To make this change permanent, add enable_experimental_vector = true to your FE configuration file (fe.conf) and restart the FE.
Key Points:
* Version Requirement: Vector indexing is only supported in shared-nothing clusters of v3.4 or later.
* Usage: Once enabled, you can create a vector index using the INDEX ... USING VECTOR syntax in your CREATE TABLE or ALTER TABLE statements.
* Search: To tune search performance (e.g., efsearch), use the ann_params hint in your query instead of a system variable:
sql
SELECT /_+ SET_VAR(ann_params='{efsearch=256}') _/ ...
...
```
If you are on a version earlier than v3.4, the vector index feature is not yet available.
References
• table_design/indexes/vector_index.mdLý Thành
07/29/2026, 3:21 AMRocky
07/29/2026, 3:22 AMCREATE TABLE statement needs a few adjustments to match the StarRocks vector index syntax (available from v3.4).
Required Changes:
1. Index Type: Use USING VECTOR and specify "index_type" = "hnsw" inside the parentheses.
2. Dimension (dim): This is a required parameter. You must specify the number of dimensions in your vector (e.g., 512, 768, 1536).
3. Metric Type: Use cosine_similarity instead of cosine.
4. Parameter Name: Change ef_construction to efconstruction (remove the underscore).
Corrected SQL:
Assuming your vectors have 1536 dimensions (common for OpenAI embeddings), here is the corrected DDL:
sql
CREATE TABLE db_rag.rag_table_store (
table_id VARCHAR(64) NOT NULL,
summary_text TEXT,
summary_vector ARRAY<FLOAT> NOT NULL,
raw_table_markdown TEXT,
-- Corrected Vector Index Definition
INDEX idx_vector (summary_vector) USING VECTOR (
"index_type" = "hnsw", -- Required: Specify HNSW here
"dim" = "1536", -- Required: Match your vector length
"metric_type" = "cosine_similarity", -- Use 'cosine_similarity' or 'l2_distance'
"M" = "16",
"efconstruction" = "40" -- Parameter name has no underscore
)
)
ENGINE=OLAP
DUPLICATE KEY(table_id)
DISTRIBUTED BY HASH(table_id);
Important Notes:
* Enable Feature: Ensure you have enabled the experimental feature first:
ADMIN SET FRONTEND CONFIG ("enable_experimental_vector" = "true");
* Data Type: ARRAY<FLOAT> is the correct type for the vector column.
* Distance Function: When querying, use approx_cosine_similarity or approx_l2_distance to leverage the index for fast approximate searches.
References
• table_design/indexes/vector_index.md
References
• table_design/indexes/vector_index.mdLý Thành
07/29/2026, 3:31 AMRocky
07/29/2026, 3:31 AMmetric_type = "cosine_similarity", you must use approx_cosine_similarity(). There is no approx_dot_product function for vector indexing.
* Selecting from Index: You cannot select from an index name (knowlegedb.idx_vector). You must select from the table name (db_rag.rag_table_store).
* Alias Placement: The AS similarity alias must be in the SELECT clause, not the WHERE clause.
* Missing Space: There is a missing space between DESC and LIMIT.
* Logic: If you only want the top 3 most similar results, you don't need a WHERE clause; the ORDER BY ... LIMIT is sufficient.
2. Corrected SQL Query:
To find the top 3 most similar rows using your vector index:
sql
SELECT
*,
approx_cosine_similarity(summary_vector, [0.015, -0.023, 0.412, 0.009]) AS similarity
FROM db_rag.rag_table_store
ORDER BY similarity DESC
LIMIT 3;
3. Requirements for Vector Search (v3.4+):
To ensure the query actually hits the vector index (ANNS):
1. Function match: Use approx_cosine_similarity (for cosine) or approx_l2_distance (for L2).
2. Order direction: Use DESC for cosine similarity and ASC for L2 distance.
3. Single column: The ORDER BY clause can only contain the vector similarity function.
4. Limit: A LIMIT clause is mandatory for approximate search.
Optional: Filtering by Score (Range Search)
If you want to find rows with a similarity score greater than 0.8 AND get the Top 3:
sql
SELECT * FROM (
SELECT
*,
approx_cosine_similarity(summary_vector, [0.015, -0.023, 0.412, 0.009]) AS similarity
FROM db_rag.rag_table_store
) t
WHERE similarity > 0.8
ORDER BY similarity DESC
LIMIT 3;
References
• table_design/indexes/vector_index.md
References
• table_design/indexes/vector_index.mdLý Thành
07/29/2026, 3:48 AMRocky
07/29/2026, 3:48 AMapprox_cosine_similarity function is used in StarRocks (starting from v3.4) to perform Approximate Nearest Neighbor Search (ANNS) using a vector index.
It is specifically designed to hit the vector index for high-speed similarity searches, whereas the standard cosine_similarity function performs a precise brute-force calculation (full table scan).
1. Syntax and Parameters
sql
approx_cosine_similarity(vector_column, constant_array)
* vector_column: The name of the column in your table that has the vector index.
* constant_array: The search vector you are comparing against (e.g., [0.1, 0.5, -0.2]). It must be a constant ARRAY<FLOAT> and match the dimension (dim) defined when the index was created.
2. Return Value
* Returns a FLOAT value between -1 and 1.
* 1.0: The vectors are identical in direction.
* 0.0: The vectors are orthogonal (90-degree angle).
* -1.0: The vectors are in opposite directions (180-degree angle).
3. Usage Requirements (To Hit the Index)
For the query to actually use the vector index (accelerated search), you must follow these rules:
1. Metric Match: Your index must have been created with "metric_type" = "cosine_similarity".
2. Order Clause: The function must be the sole expression in the ORDER BY clause.
3. Direction: You must use DESC (descending) because a higher cosine value indicates higher similarity.
4. Limit: A LIMIT clause is mandatory.
Example Query:
sql
SELECT
id,
approx_cosine_similarity(summary_vector, [0.12, 0.45, 0.88]) AS score
FROM db_rag.rag_table_store
ORDER BY score DESC
LIMIT 5;
4. Comparison: Approx vs. Precise
Feature | approx_cosine_similarity | cosine_similarity
Performance | Very Fast (uses HNSW/IVFPQ index) | Slower (Full Scan)
Accuracy | Approximate (Recall depends on params) | 100% Precise
Index Required | Yes | No
Sorting | Requires LIMIT | No restrictions
5. Tip: Normalization
If you set "is_vector_normed" = "true" when creating your index, StarRocks assumes all stored vectors and search vectors are already normalized (length = 1). This improves search performance further.
References
• table_design/indexes/vector_index.md
• sql-reference/sql-functions/math-functions/cos_similarity.md
References
• sql-reference/sql-functions/math-functions/cos_similarity.md
• sql-reference/sql-functions/math-functions/cos_similarity_norm.md
• table_design/indexes/vector_index.mdLý Thành
07/29/2026, 7:11 AMLý Thành
07/29/2026, 7:11 AMRocky
07/29/2026, 7:11 AMUS, CN).
2. VARCHAR(M)
* Type: Variable-length string.
* Range:
* v2.1 and later: 1 to 1,048,576 bytes (1 MB).
* Prior to v2.1: 1 to 65,533 bytes.
* Behavior: Only occupies storage for the actual data plus a small prefix.
* Performance Note: In some query scenarios, StarRocks pre-allocates memory based on the defined length (M) rather than the actual data length.
* Best Practice: Set M to the smallest necessary size. Do not use VARCHAR(65533) or VARCHAR(1048576) for every column, as it can significantly increase memory consumption during complex joins or aggregations.
3. STRING
* Type: Variable-length string with no explicit length limit required in the DDL.
* Range: Up to 1,048,576 bytes (1 MB).
* Equivalent to: Internally treated as VARCHAR(65533) or the system maximum depending on the version.
* Usage: Use STRING when the length of the data is unpredictable. However, for better memory efficiency, prefer VARCHAR(M) with a reasonable limit if the length is known.
──────────
Comparison Table
Feature | CHAR | VARCHAR(M) | STRING
Max Length | 255 Bytes | 1,048,576 Bytes (1 MB) | 1,048,576 Bytes (1 MB)
Unit | Bytes | Bytes | Bytes
Storage | Fixed | Variable | Variable
Memory Allocation | Based on defined length | Based on defined length | Based on max length
Primary Use Case | Very short fixed codes | Most standard text fields | Long or unknown length text
Important Considerations:
1. UTF-8 Encoding: StarRocks uses UTF-8. A single character (like a Chinese character or an Emoji) can take 3 to 4 bytes. If you are migrating from MySQL (where VARCHAR(N) counts characters), you should generally multiply your length by 3 or 4 in StarRocks.
2. Key Columns: Only CHAR and VARCHAR can be used as key columns in Aggregate or Unique key models. STRING is typically used for value columns.
References
• sql-reference/data-types/string-type/CHAR.md
• sql-reference/data-types/string-type/VARCHAR.md
• sql-reference/data-types/string-type/STRING.md
• faq/table_design_faq.md
References
• knowledge/usage/best-practice.md
• sql-reference/data-types/string-type/STRING.md
• sql-reference/data-types/string-type/VARCHAR.md
• faq/Others.md
• sql-reference/data-types/string-type/BINARY.md
• faq/table_design_faq.md
• sql-reference/System_limit.mdLý Thành
08/03/2026, 2:57 AMRocky
08/03/2026, 3:01 AMdt) must be part of the Primary Key.
* Current State: Your PK is (symbol, recorded_at), but you are partitioning by dt.
* Fix: You must add dt to the Primary Key.
2. Corrected SQL
sql
CREATE TABLE starrocks_audit_db__.fundingrate (
symbol VARCHAR(32) NOT NULL,
recorded_at BIGINT NOT NULL COMMENT 'unix ms, thời điểm script chạy',
dt DATETIME NOT NULL DEFAULT (NOW()) COMMENT 'partition key',
last_funding_rate DOUBLE NOT NULL,
mark_price DOUBLE,
index_price DOUBLE,
estimated_settle_price DOUBLE,
interest_rate DOUBLE,
next_funding_time BIGINT,
time BIGINT,
id BIGINT NOT NULL
) ENGINE=OLAP
PRIMARY KEY(symbol, recorded_at, dt) -- Added dt here
PARTITION BY date_trunc('month', dt)
DISTRIBUTED BY HASH(symbol) BUCKETS AUTO -- Recommended: use AUTO buckets
PROPERTIES (
"compression" = "LZ4",
"enable_persistent_index" = "true",
"fast_schema_evolution" = "true",
"replication_num" = "3" -- Recommended: 3 for production HA
);
3. Key Improvements & Notes
* Partition Requirement: By adding dt to the PRIMARY KEY, you satisfy the requirement that all partition columns must be subset of the primary key.
* Buckets: I changed BUCKETS 10 to BUCKETS AUTO. Unless you know your data volume is very small and constant, AUTO allows StarRocks to manage tablet size more efficiently as your data grows.
* Replication Num: You set "replication_num" = "2".
* For Production: Use 3 to ensure high availability and data safety during node failures.
* For Testing: 1 or 2 is fine.
* Database Name: starrocks_audit_db__ is a name typically reserved for internal StarRocks system audit logs. While you can use it, it is highly recommended to create your own database (e.g., funding_db) to avoid confusion with system tables.
* Column Order: In the Primary Key model, the columns making up the PK must be defined at the very beginning of the column list. I moved dt up to the 3rd position to keep the PK columns together (good for readability and ensuring no errors).
4. Logic Check: "Derived from recorded_at"
In your comment, you mentioned dt is derived from recorded_at.
* If you are inserting data via a script, you should calculate dt from the recorded_at timestamp in your application and insert it.
* If you rely on DEFAULT (NOW()), the dt will be the time the row was inserted, not necessarily when the funding rate was recorded. For a financial table, it's usually better to pass the actual time from the source.
References
• table_design/table_models/primary_key_table.md
• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
References
• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
• table_design/table_models/primary_key_table.md
• administration/management/audit_log.md
• table_design/data_distribution/Data_distribution.mdRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMRocky
08/03/2026, 3:01 AMLý Thành
08/03/2026, 3:23 AMRocky
08/03/2026, 3:24 AMUnexpected input 'AUTO' occurs because your StarRocks version (likely earlier than v2.5.7 or v3.0) does not support the AUTO keyword for bucket counts. In these versions, the parser requires an explicit integer.
Corrected DDL
Replace BUCKETS AUTO with a specific number (e.g., 10). I have also ensured dt is in the Primary Key so the partitioning works correctly.
sql
CREATE TABLE starrocks_audit_db__.fundingrate (
symbol VARCHAR(32) NOT NULL,
recorded_at BIGINT NOT NULL COMMENT 'unix ms, thời điểm script chạy',
dt DATETIME NOT NULL DEFAULT (NOW()) COMMENT 'partition key',
last_funding_rate DOUBLE NOT NULL,
mark_price DOUBLE,
index_price DOUBLE,
estimated_settle_price DOUBLE,
interest_rate DOUBLE,
next_funding_time BIGINT,
time BIGINT,
id BIGINT NOT NULL
) ENGINE=OLAP
PRIMARY KEY(symbol, recorded_at, dt)
PARTITION BY date_trunc('month', dt)
DISTRIBUTED BY HASH(symbol) BUCKETS 10 -- Use an integer here
PROPERTIES (
"compression" = "LZ4",
"enable_persistent_index" = "true",
"fast_schema_evolution" = "true",
"replication_num" = "3"
);
Why use BUCKETS 10?
Since you are partitioning by month, StarRocks creates a set of buckets for each month.
* Rule of Thumb: Aim for each bucket (tablet) to be roughly 1GB–10GB in size.
* If your funding rate data for a single month is small (e.g., a few hundred MBs or GBs), 10 buckets is more than enough.
* If you have a very large cluster (e.g., 20+ BE nodes), you might want to increase this number to ensure all nodes participate in the query, but for most financial time-series cases, 10 is a safe start.
A Note on replication_num
* If you have 3 or more Backend (BE) nodes, use "replication_num" = "3".
* If you only have 2 nodes, keep it at "2".
* If you only have 1 node (local testing), you must set it to "1".
Database Name Warning
The name starrocks_audit_db__ is very similar to StarRocks' internal system database (starrocks_monitor). It is best practice to use a custom name like trading_audit or funding_data to avoid any future conflicts with system-reserved names.
References
• table_design/data_distribution/Data_distribution.md
• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
• table_design/table_models/primary_key_table.md
References
• table_design/data_distribution/Data_distribution.md
• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
• table_design/table_models/primary_key_table.md
• knowledge/usage/best-practice.md
• administration/management/FE_configuration.md
• release_notes/release-3.3.md
• best_practices/bucketing.md
• table_design/data_distribution/feature-support-data-distribution.md
References
• table_design/data_distribution/feature-support-data-distribution.mdLý Thành
08/03/2026, 6:58 AMRocky
08/03/2026, 6:59 AMLIKE operator is case-sensitive by default. While the = operator also respects case, there are scenarios in information_schema where exact lookups (via =) might be handled by a different internal mechanism than wildcard lookups.
* Test: Try using ILIKE (case-insensitive) or `lower()`:
sql
WHERE TASK_NAME ILIKE '%KBV%'
-- OR
WHERE lower(TASK_NAME) LIKE '%kbv%'
2. Predicate Pushdown (Version Bug)
The information_schema.task_runs table is a "system view" where the Frontend (FE) handles the filtering. In some older StarRocks versions (prior to v3.1.12), there were known issues where predicates like LIKE with wildcards were not correctly pushed down or evaluated for system metadata views.
* Issue: The FE's internal "predicate lookup" for task metadata might support exact matches (=) but fail on partial matches (LIKE) due to a bug in how it scans the internal task manager.
* Fix: If you are on an older version, consider upgrading to v3.1.12+ or v3.2+, where many information_schema and task-related bugs were resolved.
3. Underscore as a Wildcard
In a LIKE pattern, the underscore _ is a wildcard that matches exactly one character. Your task name is KBV_CE.
* While LIKE '%KBV%' should still match KBV_CE, if you were trying to match the literal underscore specifically (e.g., LIKE '%KBV_%'), it would match KBVACE, KBVBCE, etc. To match a literal underscore, you must escape it: LIKE '%KBV\_%'.
4. Hidden Characters or Encoding
Since ='KBV_CE' works, the string length is likely correct. However, if the information_schema implementation in your version has a collation mismatch (e.g., the view returns utf8_bin while your session uses utf8_general_ci), LIKE can sometimes behave unexpectedly.
Recommended Next Steps:
1. Check with REGEXP: Regular expressions use a different engine and may bypass the LIKE bug:
sql
WHERE TASK_NAME REGEXP 'KBV'
2. Verify actual case: Run SELECT DISTINCT TASK_NAME FROM information_schema.task_runs to see if the name is truly uppercase.
3. Check Version: If ILIKE and REGEXP also fail, this is likely an internal bug in the FE's task metadata scanner.
References
• sql-reference/sql-functions/like-predicate-functions/like.md
• sql-reference/information_schema/task_runs.md
• release_notes/3.1.md (See bug fixes for task_runs)
References
• sql-reference/sql-functions/like-predicate-functions/like.md
• sql-reference/System_variable.md
• sql-reference/sql-statements/table_bucket_part_index/SELECT.md
• knowledge/usage/best-practice.md
• administration/management/FE_configuration.md
• sql-reference/System_limit.md
• sql-reference/sql-statements/keywords.mdLý Thành
08/03/2026, 7:44 AMtask_runs table have this issueRocky
08/03/2026, 7:44 AMinformation_schema.task_runs is a known behavior (and in many versions, a limitation/bug) in how StarRocks handles metadata views.
Why LIKE returns nothing while = works
In StarRocks, system views like task_runs and tasks are virtual tables managed entirely in the memory of the Leader FE (Frontend). They do not store data on disks like regular OLAP tables.
1. Point Lookup vs. Full Scan: When you use TASK_NAME = 'KBV_CE', the FE optimizer can perform an efficient point lookup using internal metadata maps. It knows exactly which task object to fetch.
2. Lack of Predicate Pushdown for LIKE: In many StarRocks versions (especially prior to v3.2/v3.3), the execution engine for these "SchemaScan" views is simplified. If a predicate like LIKE or REGEXP is used, the system may fail to "push down" the filter to the internal task manager. Instead of scanning all records and filtering them, the engine may return an empty set because it doesn't have a "wildcard search" implementation for the internal task history buffer.
3. Memory/RPC Limits: Task history can be massive. StarRocks limits how much metadata is returned in a single scan to prevent the FE from hanging or crashing (OOM). Partial matches often require a full scan of the task history in memory, which is sometimes blocked or results in an empty return if the "Scan" node doesn't properly iterate through the internal TaskRun objects without a specific ID.
How to solve/workaround this
If you cannot use exact matches, try the following:
* Try tasks table instead: Sometimes the information_schema.tasks (the task definitions) handles filters better than task_runs (the execution history). Find the exact name there first.
* Check Case Sensitivity: While = is often case-insensitive in MySQL-compatible clients, LIKE in StarRocks' metadata views is strictly case-sensitive. Ensure your pattern matches the case in the FE metadata exactly.
_ _Test:* WHERE TASK_NAME ILIKE '%KBV%'
* Check your Version: This specific issue (improving filter pushdown for task_runs) was a major focus in recent updates (e.g., StarRocks v3.1.12+, v3.2.5+, and v3.3+). If you are on an older version, the SchemaScan for task_runs simply doesn't support non-equality predicates for that column.
Summary
The task_runs table is not a real table; it's a window into the FE's internal memory. The code responsible for filling this window is often optimized only for point-lookups (=) to protect the FE's performance. For partial matches, the internal iterator is likely failing to populate the result set in your current version.
References
• sql-reference/information_schema/task_runs.md
• sql-reference/sql-functions/like-predicate-functions/like.md
• administration/management/FE_configuration.md
• release_notes/3.1.md
• loading/RoutineLoad.md (Context on task-task management)
References
• sql-reference/information_schema/task_runs.md
• sql-reference/sql-functions/like-predicate-functions/like.md
• administration/management/FE_configuration.md
• release_notes/3.1.md
• loading/RoutineLoad.mdRocky
08/03/2026, 7:44 AMRocky
08/03/2026, 7:44 AMLý Thành
08/03/2026, 8:18 AMRocky
08/03/2026, 8:18 AMbe_http_port) in StarRocks.
It is used by the Backend (BE) nodes for several key functions:
* Data Loading: It is used for the Stream Load process, where data is sent directly to the BE nodes via HTTP.
* Web Interface: You can access a basic web-based status page for a specific BE node by visiting http://<be_ip>:8040.
* Metrics: It provides an endpoint for monitoring tools (like Prometheus) to scrape BE-specific metrics at http://<be_ip>:8040/metrics.
* Internal Communication: Used for certain internal control tasks and log viewing via the browser.
Summary Table
Parameter | Default Port | Description
be_http_port | 8040 | BE HTTP server (Stream Load, Metrics, Status)
be_port | 9060 | BE Thrift server (FE-to-BE communication)
brpc_port | 8060 | BE bRPC server (BE-to-BE data transfer)
heartbeat_service_port | 9050 | BE heartbeat service (FE checking BE health)
References
• administration/management/BE_configuration.md
• administration/management/FE_configuration.md
References
• administration/management/BE_configuration.md
• administration/management/FE_configuration.mdLý Thành
08/03/2026, 9:36 AMRocky
08/03/2026, 9:36 AMJSON Data Type
StarRocks (v2.5+) has a native JSON type. This whole object can be stored directly in a single JSON column.
2. Timestamps
The timestamp format you are using (2026-08-03T14:40:12.253819) is supported.
* If you map these to a DATETIME column, StarRocks can handle the T separator and up to 6 digits of fractional seconds (microsecond precision) in newer versions (v3.0+).
* If you are on an older version, you might need to use str_to_date() if the automatic parsing fails.
3. Arrays
The field "product_type_id": [] is a valid empty array.
* If you are loading this into a StarRocks ARRAY<INT> column, it will be treated as an empty array.
* If you are loading into a JSON column, it will be stored as-is.
4. Null Values
Fields like "ext_number": null are handled correctly by StarRocks during loading. In a Stream Load, these will be converted to SQL NULL values in your table.
5. Loading Tip (Stream Load)
If you are loading this via Stream Load, ensure you set the header:
bash
-H "format: json" -H "strip_outer_array: true"
(Use strip_outer_array: true only if you wrap multiple of these records in []).
Does this relate to a specific error you are seeing during a StarRocks load? If so, feel free to share the error message!**Lý Thành
08/03/2026, 9:37 AMlogcall_crm_tbl (
id int(11) NOT NULL COMMENT "",
queue_id varchar(150) NULL COMMENT "",
phone_number varchar(60) NULL COMMENT "",
ext_answer varchar(60) NULL COMMENT "",
call_status varchar(150) NULL COMMENT "",
id_snowflake varchar(750) NULL COMMENT "",
id_data bigint(20) NULL COMMENT "",
campaign_id varchar(750) NULL COMMENT "",
start_time datetime NULL COMMENT "",
ringing_time int(11) NULL COMMENT "",
waiting_time int(11) NULL COMMENT "",
talking_time int(11) NULL COMMENT "",
created_at datetime NULL DEFAULT CURRENT_TIMESTAMP COMMENT "",
data json NULL COMMENT "",
update_at datetime NULL DEFAULT CURRENT_TIMESTAMP COMMENT "",
is_send varchar(765) NULL COMMENT "",
destination_caller varchar(765) NULL COMMENT "",
recording_url varchar(765) NULL COMMENT "",
ext_no_answer array<varchar(255)> NULL COMMENT ""
)
how to change data` json NULL COMMENT "" --> StringRocky
08/03/2026, 9:38 AMdata column from JSON to STRING (which in StarRocks is typically an alias for a long VARCHAR), you have two options depending on whether your table already contains data.
Option 1: If you are creating a new table
Simply replace the JSON keyword with STRING or VARCHAR(length). In StarRocks, STRING is a shorthand for VARCHAR(65533).
sql
CREATE TABLE `logcall_crm_tbl` (
...
`data` STRING NULL COMMENT "", -- Changed from JSON to STRING
...
);
Option 2: If the table already exists (with data)
StarRocks does not currently support direct type conversion from JSON to STRING using a single ALTER TABLE MODIFY COLUMN statement. You must use the "Add-Update-Drop-Rename" workflow:
1. Add a new temporary column:
sql
ALTER TABLE logcall_crm_tbl ADD COLUMN data_new STRING;
2. Copy and convert the data:
You can use the CAST function to convert JSON objects into their string representation.
sql
UPDATE logcall_crm_tbl SET data_new = CAST(data AS STRING);
3. Drop the old JSON column:
sql
ALTER TABLE logcall_crm_tbl DROP COLUMN data;
4. Rename the new column:
sql
-- Supported in v3.3.2+
ALTER TABLE logcall_crm_tbl RENAME COLUMN data_new TO data;
Important Notes:
* Storage Size: STRING / VARCHAR has a maximum length of 65,533 bytes. If your JSON objects are extremely large (MBs), you may need to use the JSON type (which supports up to 16MB) or handle them differently.
* Querying: Once converted to STRING, you can no longer use the arrow operator (data->'$.id'). You will have to use functions like get_json_string(data, '$.id').
* Wait for completion: After running ALTER TABLE (ADD/DROP/RENAME), you can check the status of the background task with:
sql
SHOW ALTER TABLE COLUMN;
References
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
• sql-reference/data-types/semi_structured/JSON.md
• sql-reference/sql-functions/json-functions/get_json_string.md
References
• using_starrocks/Flat_json.md
• sql-reference/data-types/semi_structured/JSON.mdLý Thành
08/03/2026, 9:40 AMRocky
08/03/2026, 9:40 AMDATA is not a reserved keyword in StarRocks.
You can use it as a column name directly without issues. However, it is a common word in SQL, so it is often safer to use backticks (`` ` ``) to avoid any potential ambiguity or conflict with future updates.
How to use it as a column name:
In your CREATE TABLE statement, you can define it like this:
sql
CREATE TABLE `logcall_crm_tbl` (
...
`data` STRING NULL COMMENT "", -- Using backticks is recommended
...
);
Key Rules for Column Names:
* Backticks: Always use backticks (`` column_name ``) if you want to be 100% safe, especially for names like data, order, key, or time.
* Characters: Names can include letters (a-z, A-Z), digits (0-9), and underscores (_).
* Case Sensitivity: Column names are not case-sensitive in StarRocks.
Reserved vs. Non-reserved Keywords
* Reserved Keywords: Examples include SELECT, TABLE, JOIN, PRIMARY. These must be enclosed in backticks if used as names.
* Non-reserved Keywords: Examples include DATA, DB, STATS. These can be used without backticks, but using them is still a best practice for clarity.
References
• sql-reference/sql-statements/keywords.md
• sql-reference/System_limit.md
References
• sql-reference/sql-statements/keywords.md
• sql-reference/System_limit.md
References
• faq/Sql_faq.md
• sql-reference/sql-statements/table_bucket_part_index/SELECT.md
• sql-reference/sql-statements/keywords.md
• administration/management/FE_configuration.md
• sql-reference/System_limit.md
• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
• release_notes/release-3.3.mdLý Thành
08/03/2026, 10:03 AMdebt_collection_cus_action_hist_raw SELECT knowlegedb.a.ID, knowlegedb.a.SYS_RUN_DATE,Rocky
08/03/2026, 10:03 AMINSERT OVERWRITE task while the source table is actively being updated via Kafka (Routine Load) does affect the database, primarily in terms of resource consumption.
Here is a breakdown of how it affects the system and how to manage it:
1. Resource Consumption (CPU, Memory, I/O)
* Source Table Overhead: If your source table uses the Primary Key model (common for Kafka streams with updates), reading from it during a large SELECT requires StarRocks to resolve the latest versions of rows. This consumes extra CPU and memory.
* Write Overhead: INSERT OVERWRITE writes a completely new version of the data to disk before swapping it. This creates significant disk I/O and triggers background compaction tasks on the target table to merge these new data versions.
* Network/Concurrency: If both Kafka ingestion and the INSERT OVERWRITE task are running at high frequency, they will compete for the same FE/BE resources, potentially slowing down both the ingestion speed and the query performance.
2. Why INSERT OVERWRITE is Safe (Atomicity)
The good news is that INSERT OVERWRITE is atomic.
* StarRocks creates a temporary partition/table for the new data.
* The old data remains available for queries while the task is running.
* Only when the task succeeds does StarRocks "swap" the old data with the new.
* Result: Your users will never see "partial" data or an empty table during the process.
3. Impact of "Many Updates"
If the source table has "many updates," it means the background compaction on that source table is already working hard. Adding a heavy INSERT OVERWRITE query adds more pressure. If the cluster is already near 80-90% CPU usage, this task could cause query latency spikes.
──────────
Best Practices & Recommendations
A. Use Asynchronous Materialized Views (Recommended)
Instead of a manual INSERT OVERWRITE task, use an Asynchronous Materialized View.
* Why: Async MVs can perform incremental refreshes (at the partition level). If only one day of data changed in your Kafka source, StarRocks only refreshes that one partition in the MV, rather than overwriting the whole table.
* Syntax:
sql
CREATE MATERIALIZED VIEW my_mv
REFRESH ASYNC [EVERY INTERVAL]
AS SELECT ... FROM source_table;
B. Partition-Level Overwrite
If you must use INSERT OVERWRITE, avoid overwriting the entire table if it is large. If your table is partitioned by date, overwrite only the affected partitions:
sql
INSERT OVERWRITE `debt_collection_cus_action_hist_raw`
PARTITION (p20231027)
SELECT ... FROM ... WHERE date = '2023-10-27';
C. Control Frequency
Do not run INSERT OVERWRITE too frequently (e.g., every minute). This creates too many data versions, which will eventually slow down the database. For near-real-time needs, the Materialized View approach is much more efficient.
D. Check Task Status
You can monitor the performance and status of your task using:
sql
SELECT * FROM information_schema.task_runs
WHERE task_name = 'your_task_name'
ORDER BY create_time DESC;
References
• loading/InsertInto.md
• using_starrocks/async_mv/Materialized_view.md
• faq/loading/Routine_load_faq.md
References
• faq/loading/Routine_load_faq.md
• loading/RoutineLoad.md