l
@Rocky hello
r
Hello! I'm Rocky, your StarRocks assistant. How can I help you today?
l
@Rocky SQL Error [1193] [HY000]: Getting analyzing error. Detail message: Unknown system variable 'enable_vector_index', the most similar variables are {'enable_dialect_downgrade', 'enable_tde', 'enable_group_execution'}.
r
The error occurs because
enable_vector_index
is not a valid system variable in StarRocks. Starting from v3.4, vector indexing is supported as a beta feature. To enable it, you must use the FE configuration item
enable_experimental_vector
. How to enable vector indexing You can enable it dynamically using the following command:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("enable_experimental_vector" = "true");
To make this change permanent, add
enable_experimental_vector = true
to your FE configuration file (
fe.conf
) and restart the FE. Key Points: * Version Requirement: Vector indexing is only supported in shared-nothing clusters of v3.4 or later. * Usage: Once enabled, you can create a vector index using the
INDEX ... USING VECTOR
syntax in your
CREATE TABLE
or
ALTER TABLE
statements. * Search: To tune search performance (e.g.,
efsearch
), use the
ann_params
hint in your query instead of a system variable:
Copy code
sql
    SELECT /_+ SET_VAR(ann_params='{efsearch=256}') _/ ...
... ``` If you are on a version earlier than v3.4, the vector index feature is not yet available. References • table_design/indexes/vector_index.md
l
@Rocky CREATE TABLE db_rag.rag_table_store ( table_id VARCHAR(64) NOT NULL, summary_text TEXT, summary_vector ARRAY<FLOAT> NOT NULL, -- Lưu chuỗi vector số thực raw_table_markdown TEXT, -- Định nghĩa chỉ mục Vector INDEX idx_vector (summary_vector) USING HNSW ( "metric_type" = "cosine", -- Định dạng khoảng cách (cosine hoặc l2) "M" = "16", -- Tham số liên kết đồ thị HNSW "ef_construction" = "40" -- Tham số tối ưu khi dựng index ) ) ENGINE=OLAP DUPLICATE KEY(table_id) DISTRIBUTED BY HASH(table_id); correct?
r
Your
CREATE TABLE
statement needs a few adjustments to match the StarRocks vector index syntax (available from v3.4). Required Changes: 1. Index Type: Use
USING VECTOR
and specify
"index_type" = "hnsw"
inside the parentheses. 2. Dimension (
dim
)
: This is a required parameter. You must specify the number of dimensions in your vector (e.g., 512, 768, 1536). 3. Metric Type: Use
cosine_similarity
instead of
cosine
. 4. Parameter Name: Change
ef_construction
to
efconstruction
(remove the underscore). Corrected SQL: Assuming your vectors have 1536 dimensions (common for OpenAI embeddings), here is the corrected DDL:
Copy code
sql
CREATE TABLE db_rag.rag_table_store (
  table_id VARCHAR(64) NOT NULL,
  summary_text TEXT,
  summary_vector ARRAY<FLOAT> NOT NULL, 
  raw_table_markdown TEXT,

  -- Corrected Vector Index Definition
  INDEX idx_vector (summary_vector) USING VECTOR (
    "index_type" = "hnsw",        -- Required: Specify HNSW here
    "dim" = "1536",               -- Required: Match your vector length
    "metric_type" = "cosine_similarity", -- Use 'cosine_similarity' or 'l2_distance'
    "M" = "16",
    "efconstruction" = "40"       -- Parameter name has no underscore
  )
)
ENGINE=OLAP
DUPLICATE KEY(table_id)
DISTRIBUTED BY HASH(table_id);
Important Notes: * Enable Feature: Ensure you have enabled the experimental feature first:
ADMIN SET FRONTEND CONFIG ("enable_experimental_vector" = "true");
* Data Type:
ARRAY<FLOAT>
is the correct type for the vector column. * Distance Function: When querying, use
approx_cosine_similarity
or
approx_l2_distance
to leverage the index for fast approximate searches. References • table_design/indexes/vector_index.md References • table_design/indexes/vector_index.md
l
@Rocky SELECT * from rag_table_store where approx_dot_product(summary_vector, [0.015, -0.023, 0.412, ..., 0.009]) AS similarity FROM knowlegedb.idx_vector ORDER BY similarity DESCLIMIT 3; review this sql
r
Your SQL query contains several syntax and logic errors. Here is the review and the corrected version. 1. Key Issues Found: * Wrong Function: For a vector index with
metric_type = "cosine_similarity"
, you must use
approx_cosine_similarity()
. There is no
approx_dot_product
function for vector indexing. * Selecting from Index: You cannot select from an index name (
knowlegedb.idx_vector
). You must select from the table name (
db_rag.rag_table_store
). * Alias Placement: The
AS similarity
alias must be in the
SELECT
clause, not the
WHERE
clause. * Missing Space: There is a missing space between
DESC
and
LIMIT
. * Logic: If you only want the top 3 most similar results, you don't need a
WHERE
clause; the
ORDER BY ... LIMIT
is sufficient. 2. Corrected SQL Query: To find the top 3 most similar rows using your vector index:
Copy code
sql
SELECT 
    *, 
    approx_cosine_similarity(summary_vector, [0.015, -0.023, 0.412, 0.009]) AS similarity
FROM db_rag.rag_table_store
ORDER BY similarity DESC
LIMIT 3;
3. Requirements for Vector Search (v3.4+): To ensure the query actually hits the vector index (ANNS): 1. Function match: Use
approx_cosine_similarity
(for cosine) or
approx_l2_distance
(for L2). 2. Order direction: Use
DESC
for cosine similarity and
ASC
for L2 distance. 3. Single column: The
ORDER BY
clause can only contain the vector similarity function. 4. Limit: A
LIMIT
clause is mandatory for approximate search. Optional: Filtering by Score (Range Search) If you want to find rows with a similarity score greater than
0.8
AND get the Top 3:
Copy code
sql
SELECT * FROM (
    SELECT 
        *, 
        approx_cosine_similarity(summary_vector, [0.015, -0.023, 0.412, 0.009]) AS similarity
    FROM db_rag.rag_table_store
) t
WHERE similarity > 0.8
ORDER BY similarity DESC
LIMIT 3;
References • table_design/indexes/vector_index.md References • table_design/indexes/vector_index.md
l
@Rocky approx_cosine_similarity detail about this fundtion
r
The
approx_cosine_similarity
function is used in StarRocks (starting from v3.4) to perform Approximate Nearest Neighbor Search (ANNS) using a vector index. It is specifically designed to hit the vector index for high-speed similarity searches, whereas the standard
cosine_similarity
function performs a precise brute-force calculation (full table scan). 1. Syntax and Parameters
Copy code
sql
approx_cosine_similarity(vector_column, constant_array)
*
vector_column
: The name of the column in your table that has the vector index. *
constant_array
: The search vector you are comparing against (e.g.,
[0.1, 0.5, -0.2]
). It must be a constant
ARRAY<FLOAT>
and match the dimension (
dim
) defined when the index was created. 2. Return Value * Returns a FLOAT value between -1 and 1. * 1.0: The vectors are identical in direction. * 0.0: The vectors are orthogonal (90-degree angle). * -1.0: The vectors are in opposite directions (180-degree angle). 3. Usage Requirements (To Hit the Index) For the query to actually use the vector index (accelerated search), you must follow these rules: 1. Metric Match: Your index must have been created with
"metric_type" = "cosine_similarity"
. 2. Order Clause: The function must be the sole expression in the
ORDER BY
clause. 3. Direction: You must use
DESC
(descending) because a higher cosine value indicates higher similarity. 4. Limit: A
LIMIT
clause is mandatory. Example Query:
Copy code
sql
SELECT 
    id, 
    approx_cosine_similarity(summary_vector, [0.12, 0.45, 0.88]) AS score
FROM db_rag.rag_table_store
ORDER BY score DESC
LIMIT 5;
4. Comparison: Approx vs. Precise Feature |
approx_cosine_similarity
|
cosine_similarity
Performance | Very Fast (uses HNSW/IVFPQ index) | Slower (Full Scan) Accuracy | Approximate (Recall depends on params) | 100% Precise Index Required | Yes | No Sorting | Requires
LIMIT
| No restrictions 5. Tip: Normalization If you set
"is_vector_normed" = "true"
when creating your index, StarRocks assumes all stored vectors and search vectors are already normalized (length = 1). This improves search performance further. References • table_design/indexes/vector_index.md • sql-reference/sql-functions/math-functions/cos_similarity.md References • sql-reference/sql-functions/math-functions/cos_similarity.md • sql-reference/sql-functions/math-functions/cos_similarity_norm.md • table_design/indexes/vector_index.md
l
detail data type for string, varchar, char
@Rocky detail data type for string, varchar, char
r
In StarRocks, string types are defined by their length in bytes (not characters) and how they handle memory during query execution. 1. CHAR * Type: Fixed-length string. * Range: 1 to 255 bytes. * Behavior: If the stored string is shorter than the defined length, it is padded with spaces. * Usage: Best for very short, fixed-length identifiers (e.g., country codes like
US
,
CN
). 2. VARCHAR(M) * Type: Variable-length string. * Range: * v2.1 and later: 1 to 1,048,576 bytes (1 MB). * Prior to v2.1: 1 to 65,533 bytes. * Behavior: Only occupies storage for the actual data plus a small prefix. * Performance Note: In some query scenarios, StarRocks pre-allocates memory based on the defined length (M) rather than the actual data length. * Best Practice: Set
M
to the smallest necessary size. Do not use
VARCHAR(65533)
or
VARCHAR(1048576)
for every column, as it can significantly increase memory consumption during complex joins or aggregations. 3. STRING * Type: Variable-length string with no explicit length limit required in the DDL. * Range: Up to 1,048,576 bytes (1 MB). * Equivalent to: Internally treated as
VARCHAR(65533)
or the system maximum depending on the version. * Usage: Use
STRING
when the length of the data is unpredictable. However, for better memory efficiency, prefer
VARCHAR(M)
with a reasonable limit if the length is known. ────────── Comparison Table Feature | CHAR | VARCHAR(M) | STRING Max Length | 255 Bytes | 1,048,576 Bytes (1 MB) | 1,048,576 Bytes (1 MB) Unit | Bytes | Bytes | Bytes Storage | Fixed | Variable | Variable Memory Allocation | Based on defined length | Based on defined length | Based on max length Primary Use Case | Very short fixed codes | Most standard text fields | Long or unknown length text Important Considerations: 1. UTF-8 Encoding: StarRocks uses UTF-8. A single character (like a Chinese character or an Emoji) can take 3 to 4 bytes. If you are migrating from MySQL (where
VARCHAR(N)
counts characters), you should generally multiply your length by 3 or 4 in StarRocks. 2. Key Columns: Only
CHAR
and
VARCHAR
can be used as key columns in Aggregate or Unique key models.
STRING
is typically used for value columns. References • sql-reference/data-types/string-type/CHAR.md • sql-reference/data-types/string-type/VARCHAR.md • sql-reference/data-types/string-type/STRING.md • faq/table_design_faq.md References • knowledge/usage/best-practice.md • sql-reference/data-types/string-type/STRING.md • sql-reference/data-types/string-type/VARCHAR.md • faq/Others.md • sql-reference/data-types/string-type/BINARY.md • faq/table_design_faq.md • sql-reference/System_limit.md
l
@Rocky CREATE TABLE starrocks_audit_db__.fundingrate ( symbol *VARCHAR*(32) NOT NULL, recorded_at BIGINT NOT NULL COMMENT 'unix ms, thời điểm script chạy', last_funding_rate DOUBLE NOT NULL, mark_price DOUBLE, index_price DOUBLE, estimated_settle_price DOUBLE, interest_rate DOUBLE, next_funding_time BIGINT, time BIGINT, dt DATETIME DEFAULT (*NOW*()) COMMENT 'partition key, derived from recorded_at', id BIGINT NOT NULL ) ENGINE=OLAP PRIMARY *KEY*(symbol, recorded_at) PARTITION BY date_trunc('month', dt) DISTRIBUTED BY HASH(symbol) BUCKETS 10 PROPERTIES ( "compression" = "LZ4", "enable_persistent_index" = "true", "fast_schema_evolution" = "true", "replicated_storage" = "true", "replication_num" = "2" ); correct?
r
Your DDL is almost correct but has one critical error regarding the Primary Key and Partitioning, along with a few best-practice recommendations. 1. The Critical Error: Partition Column vs. Primary Key In StarRocks, if you use a Primary Key model, the columns used for partitioning (in your case,
dt
) must be part of the Primary Key. * Current State: Your PK is
(symbol, recorded_at)
, but you are partitioning by
dt
. * Fix: You must add
dt
to the Primary Key. 2. Corrected SQL
Copy code
sql
CREATE TABLE starrocks_audit_db__.fundingrate (
    symbol                  VARCHAR(32)  NOT NULL,
    recorded_at             BIGINT       NOT NULL COMMENT 'unix ms, thời điểm script chạy',
    dt                      DATETIME     NOT NULL DEFAULT (NOW()) COMMENT 'partition key',
    last_funding_rate       DOUBLE       NOT NULL,
    mark_price              DOUBLE,
    index_price             DOUBLE,
    estimated_settle_price  DOUBLE,
    interest_rate           DOUBLE,
    next_funding_time       BIGINT,
    time                    BIGINT,
    id                      BIGINT       NOT NULL
) ENGINE=OLAP
PRIMARY KEY(symbol, recorded_at, dt) -- Added dt here
PARTITION BY date_trunc('month', dt)
DISTRIBUTED BY HASH(symbol) BUCKETS AUTO -- Recommended: use AUTO buckets
PROPERTIES (
    "compression" = "LZ4",
    "enable_persistent_index" = "true",
    "fast_schema_evolution" = "true",
    "replication_num" = "3" -- Recommended: 3 for production HA
);
3. Key Improvements & Notes * Partition Requirement: By adding
dt
to the
PRIMARY KEY
, you satisfy the requirement that all partition columns must be subset of the primary key. * Buckets: I changed
BUCKETS 10
to
BUCKETS AUTO
. Unless you know your data volume is very small and constant,
AUTO
allows StarRocks to manage tablet size more efficiently as your data grows. * Replication Num: You set
"replication_num" = "2"
. * For Production: Use
3
to ensure high availability and data safety during node failures. * For Testing:
1
or
2
is fine. * Database Name:
starrocks_audit_db__
is a name typically reserved for internal StarRocks system audit logs. While you can use it, it is highly recommended to create your own database (e.g.,
funding_db
) to avoid confusion with system tables. * Column Order: In the Primary Key model, the columns making up the PK must be defined at the very beginning of the column list. I moved
dt
up to the 3rd position to keep the PK columns together (good for readability and ensuring no errors). 4. Logic Check: "Derived from recorded_at" In your comment, you mentioned
dt
is derived from
recorded_at
. * If you are inserting data via a script, you should calculate
dt
from the
recorded_at
timestamp in your application and insert it. * If you rely on
DEFAULT (NOW())
, the
dt
will be the time the row was inserted, not necessarily when the funding rate was recorded. For a financial table, it's usually better to pass the actual time from the source. References • table_design/table_models/primary_key_table.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md References • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • table_design/table_models/primary_key_table.md • administration/management/audit_log.md • table_design/data_distribution/Data_distribution.md
• table_design/data_types/BIGINT.md • table_design/data_types/DOUBLE.md • table_design/data_types/DATETIME.md • table_design/data_types/VARCHAR.md • table_design/data_distribution/automatic_bucketing.md • table_design/data_distribution/expression_partitioning.md • table_design/data_distribution/list_partitioning.md • sql-reference/sql-functions/date-time-functions/date_trunc.md • sql-reference/sql-functions/date-time-functions/now.md • table_design/data_types/Scalar_types.md • table_design/Storage_compression.md • administration/Configuration.md • table_design/Sort_key.md • table_design/Best_practice_table_design.md • table_design/table_models/Aggregate_table.md • table_design/table_models/Duplicate_key_table.md • table_design/table_models/Unique_table.md • loading/loading_introduction/loading_introduction.md • loading/StreamLoad.md • loading/RoutineLoad.md • administration/Management.md • administration/Replica_management.md • benchmarking/SSB_Benchmark.md • benchmarking/TPCH_Benchmark.md • introduction/StarRocks_intro.md • introduction/Features.md • quick_start/StarRocks_intro.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md • sql-reference/sql-statements/table_bucket_part_index/TRUNCATE_TABLE.md • sql-reference/sql-statements/data-manipulation/INSERT.md • sql-reference/sql-statements/data-manipulation/UPDATE.md • sql-reference/sql-statements/data-manipulation/DELETE.md
• sql-reference/sql-functions/aggregate-functions/count.md • sql-reference/sql-functions/aggregate-functions/sum.md • sql-reference/sql-functions/aggregate-functions/avg.md • sql-reference/sql-functions/aggregate-functions/max.md • sql-reference/sql-functions/aggregate-functions/min.md • using_starrocks/Query_planning.md • using_starrocks/CBO.md • using_starrocks/Colocate_join.md • using_starrocks/Join_optimization.md • using_starrocks/Materialized_view.md • using_starrocks/Partition_pruning.md • using_starrocks/Index.md • using_starrocks/Bitmap_index.md • using_starrocks/Bloom_filter_index.md • using_starrocks/Vector_index.md • using_starrocks/StarRocks_architecture.md • deployment/Deployment_overview.md • deployment/Standalone_deployment.md • deployment/Distributed_deployment.md • deployment/Docker_deployment.md • deployment/K8s_deployment.md • administration/Monitor_and_Alert.md • administration/Troubleshooting.md • administration/Backup_and_restore.md • administration/Upgrade_StarRocks.md • administration/User_privilege.md • administration/Resource_management.md • reference/System_variables.md • reference/Information_schema.md • reference/Show_commands.md • reference/Explain_plan.md • integrations/Integrations_overview.md • integrations/Spark_connector.md • integrations/Flink_connector.md • integrations/Datax_starrocks_writer.md • integrations/Dbt_starrocks_adapter.md • integrations/Superset_integration.md
• sql-reference/sql-functions/Numeric_functions.md • sql-reference/sql-functions/Date_time_functions.md • sql-reference/sql-functions/Aggregate_functions.md • sql-reference/sql-functions/Window_functions.md • sql-reference/sql-functions/Bitmap_functions.md • sql-reference/sql-functions/HLL_functions.md • sql-reference/sql-functions/JSON_functions.md • sql-reference/sql-functions/Array_functions.md • sql-reference/sql-functions/Map_functions.md • sql-reference/sql-functions/Struct_functions.md • sql-reference/sql-functions/Lambda_functions.md • sql-reference/sql-functions/Cast_functions.md • sql-reference/sql-functions/Conditional_functions.md • sql-reference/sql-functions/Encryption_functions.md • sql-reference/sql-functions/Information_functions.md • sql-reference/sql-functions/Mathematical_functions.md • sql-reference/sql-functions/Table_functions.md • sql-reference/sql-functions/Other_functions.md • sql-reference/sql-statements/Account_management.md • sql-reference/sql-statements/Cluster_management.md • sql-reference/sql-statements/Data_definition.md • sql-reference/sql-statements/Data_manipulation.md • sql-reference/sql-statements/Data_query.md • sql-reference/sql-statements/Privilege_management.md • sql-reference/sql-statements/Resource_management.md • sql-reference/sql-statements/Utility.md • sql-reference/sql-statements/Variable_management.md • sql-reference/sql-statements/Materialized_view.md • sql-reference/sql-statements/External_catalog.md • sql-reference/sql-statements/External_table.md
• sql-reference/sql-statements/System_management.md • sql-reference/sql-statements/Transaction.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md
• sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md
• sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md
• sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md
• sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md • sql-reference/sql-statements/Function.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Account.md • sql-reference/sql-statements/Privilege.md • sql-reference/sql-statements/Resource.md • sql-reference/sql-statements/System.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • sql-reference/sql-statements/Dictionary.md • sql-reference/sql-statements/Pipe.md • sql-reference/sql-statements/Storage_volume.md • sql-reference/sql-statements/Workload_group.md • sql-reference/sql-statements/Index.md • sql-reference/sql-statements/Table_bucket_part_index.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/View.md
l
@Rocky Error occurred during SQL query execution SQL Error [1064] [42000]: Getting syntax error at line 16, column 36. Detail message: Unexpected input 'AUTO', the most similar input is {INTEGER_VALUE}.
r
The error
Unexpected input 'AUTO'
occurs because your StarRocks version (likely earlier than v2.5.7 or v3.0) does not support the
AUTO
keyword for bucket counts. In these versions, the parser requires an explicit integer. Corrected DDL Replace
BUCKETS AUTO
with a specific number (e.g.,
10
). I have also ensured
dt
is in the Primary Key so the partitioning works correctly.
Copy code
sql
CREATE TABLE starrocks_audit_db__.fundingrate (
    symbol                  VARCHAR(32)  NOT NULL,
    recorded_at             BIGINT       NOT NULL COMMENT 'unix ms, thời điểm script chạy',
    dt                      DATETIME     NOT NULL DEFAULT (NOW()) COMMENT 'partition key',
    last_funding_rate       DOUBLE       NOT NULL,
    mark_price              DOUBLE,
    index_price             DOUBLE,
    estimated_settle_price  DOUBLE,
    interest_rate           DOUBLE,
    next_funding_time       BIGINT,
    time                    BIGINT,
    id                      BIGINT       NOT NULL
) ENGINE=OLAP
PRIMARY KEY(symbol, recorded_at, dt)
PARTITION BY date_trunc('month', dt)
DISTRIBUTED BY HASH(symbol) BUCKETS 10 -- Use an integer here
PROPERTIES (
    "compression" = "LZ4",
    "enable_persistent_index" = "true",
    "fast_schema_evolution" = "true",
    "replication_num" = "3" 
);
Why use
BUCKETS 10
?
Since you are partitioning by month, StarRocks creates a set of buckets for each month. * Rule of Thumb: Aim for each bucket (tablet) to be roughly 1GB–10GB in size. * If your funding rate data for a single month is small (e.g., a few hundred MBs or GBs), 10 buckets is more than enough. * If you have a very large cluster (e.g., 20+ BE nodes), you might want to increase this number to ensure all nodes participate in the query, but for most financial time-series cases, 10 is a safe start. A Note on
replication_num
* If you have 3 or more Backend (BE) nodes, use
"replication_num" = "3"
. * If you only have 2 nodes, keep it at
"2"
. * If you only have 1 node (local testing), you must set it to
"1"
. Database Name Warning The name
starrocks_audit_db__
is very similar to StarRocks' internal system database (
starrocks_monitor
). It is best practice to use a custom name like
trading_audit
or
funding_data
to avoid any future conflicts with system-reserved names. References • table_design/data_distribution/Data_distribution.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • table_design/table_models/primary_key_table.md References • table_design/data_distribution/Data_distribution.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • table_design/table_models/primary_key_table.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • release_notes/release-3.3.md • best_practices/bucketing.md • table_design/data_distribution/feature-support-data-distribution.md References • table_design/data_distribution/feature-support-data-distribution.md
l
@Rocky why a.TASK_NAME ='KBV_CE' can run but TASK_NAME LIKE '%KBV%' not return anything SELECT * FROM information_schema.task_runs a WHERE a.TASK_NAME ='KBV_CE' SELECT * FROM information_schema.task_runs WHERE TASK_NAME LIKE '%KBV%'
r
This behavior usually occurs for one of the following reasons in StarRocks: 1. Case Sensitivity In StarRocks, the
LIKE
operator is case-sensitive by default. While the
=
operator also respects case, there are scenarios in
information_schema
where exact lookups (via
=
) might be handled by a different internal mechanism than wildcard lookups. * Test: Try using
ILIKE
(case-insensitive) or `lower()`:
Copy code
sql
    WHERE TASK_NAME ILIKE '%KBV%'
    -- OR
    WHERE lower(TASK_NAME) LIKE '%kbv%'
2. Predicate Pushdown (Version Bug) The
information_schema.task_runs
table is a "system view" where the Frontend (FE) handles the filtering. In some older StarRocks versions (prior to v3.1.12), there were known issues where predicates like
LIKE
with wildcards were not correctly pushed down or evaluated for system metadata views. * Issue: The FE's internal "predicate lookup" for task metadata might support exact matches (
=
) but fail on partial matches (
LIKE
) due to a bug in how it scans the internal task manager. * Fix: If you are on an older version, consider upgrading to v3.1.12+ or v3.2+, where many
information_schema
and task-related bugs were resolved. 3. Underscore as a Wildcard In a
LIKE
pattern, the underscore
_
is a wildcard that matches exactly one character. Your task name is
KBV_CE
. * While
LIKE '%KBV%'
should still match
KBV_CE
, if you were trying to match the literal underscore specifically (e.g.,
LIKE '%KBV_%'
), it would match
KBVACE
,
KBVBCE
, etc. To match a literal underscore, you must escape it:
LIKE '%KBV\_%'
. 4. Hidden Characters or Encoding Since
='KBV_CE'
works, the string length is likely correct. However, if the
information_schema
implementation in your version has a collation mismatch (e.g., the view returns
utf8_bin
while your session uses
utf8_general_ci
),
LIKE
can sometimes behave unexpectedly. Recommended Next Steps: 1. Check with
REGEXP
: Regular expressions use a different engine and may bypass the
LIKE
bug:
Copy code
sql
    WHERE TASK_NAME REGEXP 'KBV'
2. Verify actual case: Run
SELECT DISTINCT TASK_NAME FROM information_schema.task_runs
to see if the name is truly uppercase. 3. Check Version: If
ILIKE
and
REGEXP
also fail, this is likely an internal bug in the FE's task metadata scanner. References • sql-reference/sql-functions/like-predicate-functions/like.md • sql-reference/information_schema/task_runs.md • release_notes/3.1.md (See bug fixes for
task_runs
) References • sql-reference/sql-functions/like-predicate-functions/like.md • sql-reference/System_variable.md • sql-reference/sql-statements/table_bucket_part_index/SELECT.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • sql-reference/System_limit.md • sql-reference/sql-statements/keywords.md
l
@Rocky not work, only = return data, why
task_runs table have this issue
r
The issue you are encountering with
information_schema.task_runs
is a known behavior (and in many versions, a limitation/bug) in how StarRocks handles metadata views. Why
LIKE
returns nothing while
=
works
In StarRocks, system views like
task_runs
and
tasks
are virtual tables managed entirely in the memory of the Leader FE (Frontend). They do not store data on disks like regular OLAP tables. 1. Point Lookup vs. Full Scan: When you use
TASK_NAME = 'KBV_CE'
, the FE optimizer can perform an efficient point lookup using internal metadata maps. It knows exactly which task object to fetch. 2. Lack of Predicate Pushdown for
LIKE
: In many StarRocks versions (especially prior to v3.2/v3.3), the execution engine for these "SchemaScan" views is simplified. If a predicate like
LIKE
or
REGEXP
is used, the system may fail to "push down" the filter to the internal task manager. Instead of scanning all records and filtering them, the engine may return an empty set because it doesn't have a "wildcard search" implementation for the internal task history buffer. 3. Memory/RPC Limits: Task history can be massive. StarRocks limits how much metadata is returned in a single scan to prevent the FE from hanging or crashing (OOM). Partial matches often require a full scan of the task history in memory, which is sometimes blocked or results in an empty return if the "Scan" node doesn't properly iterate through the internal
TaskRun
objects without a specific ID. How to solve/workaround this If you cannot use exact matches, try the following: * Try
tasks
table instead
: Sometimes the
information_schema.tasks
(the task definitions) handles filters better than
task_runs
(the execution history). Find the exact name there first. * Check Case Sensitivity: While
=
is often case-insensitive in MySQL-compatible clients,
LIKE
in StarRocks' metadata views is strictly case-sensitive. Ensure your pattern matches the case in the FE metadata exactly. _ _Test:*
WHERE TASK_NAME ILIKE '%KBV%'
* Check your Version: This specific issue (improving filter pushdown for
task_runs
) was a major focus in recent updates (e.g., StarRocks v3.1.12+, v3.2.5+, and v3.3+). If you are on an older version, the
SchemaScan
for
task_runs
simply doesn't support non-equality predicates for that column. Summary The
task_runs
table is not a real table; it's a window into the FE's internal memory. The code responsible for filling this window is often optimized only for point-lookups (
=
) to protect the FE's performance. For partial matches, the internal iterator is likely failing to populate the result set in your current version. References • sql-reference/information_schema/task_runs.md • sql-reference/sql-functions/like-predicate-functions/like.md • administration/management/FE_configuration.md • release_notes/3.1.md • loading/RoutineLoad.md (Context on task-task management) References • sql-reference/information_schema/task_runs.md • sql-reference/sql-functions/like-predicate-functions/like.md • administration/management/FE_configuration.md • release_notes/3.1.md • loading/RoutineLoad.md
• sql-reference/sql-statements/table_bucket_part_index/SELECT.md • sql-reference/System_variable.md • sql-reference/sql-statements/keywords.md • knowledge/usage/best-practice.md • sql-reference/System_limit.md • using_starrocks/Query_planning.md • using_starrocks/CBO.md • using_starrocks/Partition_pruning.md • table_design/data_distribution/Data_distribution.md • table_design/Best_practice_table_design.md • administration/Monitor_and_Alert.md • administration/Troubleshooting.md • faq/Query_FAQ.md • faq/Others_FAQ.md • reference/Information_schema.md • reference/Show_commands.md • reference/Explain_plan.md • sql-reference/sql-functions/Date_time_functions.md • sql-reference/sql-functions/String_functions.md • sql-reference/sql-functions/Aggregate_functions.md • sql-reference/sql-functions/Conditional_functions.md • sql-reference/sql-functions/Information_functions.md • sql-reference/sql-functions/Other_functions.md • sql-reference/sql-statements/Data_query.md • sql-reference/sql-statements/Utility.md • sql-reference/sql-statements/Variable_management.md • sql-reference/sql-statements/Materialized_view.md • sql-reference/sql-statements/Database.md • sql-reference/sql-statements/Table.md • sql-reference/sql-statements/Load.md • sql-reference/sql-statements/Query.md • sql-reference/sql-statements/Variable.md • sql-reference/sql-statements/Plan.md • administration/Management.md • introduction/Features.md • quick_start/StarRocks_intro.md
l
@Rocky port 8040 used for what
r
Port 8040 is the default BE HTTP Server Port (
be_http_port
) in StarRocks. It is used by the Backend (BE) nodes for several key functions: * Data Loading: It is used for the Stream Load process, where data is sent directly to the BE nodes via HTTP. * Web Interface: You can access a basic web-based status page for a specific BE node by visiting
http://<be_ip>:8040
. * Metrics: It provides an endpoint for monitoring tools (like Prometheus) to scrape BE-specific metrics at
http://<be_ip>:8040/metrics
. * Internal Communication: Used for certain internal control tasks and log viewing via the browser. Summary Table Parameter | Default Port | Description
be_http_port
| 8040 | BE HTTP server (Stream Load, Metrics, Status)
be_port
| 9060 | BE Thrift server (FE-to-BE communication)
brpc_port
| 8060 | BE bRPC server (BE-to-BE data transfer)
heartbeat_service_port
| 9050 | BE heartbeat service (FE checking BE health) References • administration/management/BE_configuration.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md
l
@Rocky this json correct? {"id": 6384088, "flag": 1, "active": true, "lead_id": 18081758, "menu_id": 323, "mobiles": "0392251896", "action_id": 388, "call_ratio": 10.0, "created_at": "2026-08-03T144012.253819", "ext_number": null, "retry_time": 18000, "campaign_id": "SUBCAMP0000000000024", "customer_id": null, "next_action": "", "max_attempts": 2, "max_channels": 100, "phone_number": "0392251896", "queue_number": "8105", "updated_date": "2026-08-03T144012.758644", "call_attempts": 1, "callback_time": "2026-08-03T144012.253819", "last_note_code": null, "product_type_id": [], "last_call_status": null, "product_offer_id": 314608, "data_auto_processed": null, "last_time_note_code": null, "product_holding_days": 0}
r
Yes, this JSON is syntactically correct and follows standard JSON formatting. If you are planning to use this data with StarRocks, here are a few things to keep in mind: 1. Compatibility with
JSON
Data Type
StarRocks (v2.5+) has a native
JSON
type. This whole object can be stored directly in a single
JSON
column. 2. Timestamps The timestamp format you are using (
2026-08-03T14:40:12.253819
) is supported. * If you map these to a
DATETIME
column, StarRocks can handle the
T
separator and up to 6 digits of fractional seconds (microsecond precision) in newer versions (v3.0+). * If you are on an older version, you might need to use
str_to_date()
if the automatic parsing fails. 3. Arrays The field
"product_type_id": []
is a valid empty array. * If you are loading this into a StarRocks
ARRAY<INT>
column, it will be treated as an empty array. * If you are loading into a
JSON
column, it will be stored as-is. 4. Null Values Fields like
"ext_number": null
are handled correctly by StarRocks during loading. In a Stream Load, these will be converted to SQL
NULL
values in your table. 5. Loading Tip (Stream Load) If you are loading this via Stream Load, ensure you set the header:
Copy code
bash
-H "format: json" -H "strip_outer_array: true"
(Use
strip_outer_array: true
only if you wrap multiple of these records in
[]
). Does this relate to a specific error you are seeing during a StarRocks load? If so, feel free to share the error message!**
l
@Rocky CREATE TABLE
logcall_crm_tbl
(
id
int(11) NOT NULL COMMENT "",
queue_id
varchar(150) NULL COMMENT "",
phone_number
varchar(60) NULL COMMENT "",
ext_answer
varchar(60) NULL COMMENT "",
call_status
varchar(150) NULL COMMENT "",
id_snowflake
varchar(750) NULL COMMENT "",
id_data
bigint(20) NULL COMMENT "",
campaign_id
varchar(750) NULL COMMENT "",
start_time
datetime NULL COMMENT "",
ringing_time
int(11) NULL COMMENT "",
waiting_time
int(11) NULL COMMENT "",
talking_time
int(11) NULL COMMENT "",
created_at
datetime NULL DEFAULT CURRENT_TIMESTAMP COMMENT "",
data
json NULL COMMENT "",
update_at
datetime NULL DEFAULT CURRENT_TIMESTAMP COMMENT "",
is_send
varchar(765) NULL COMMENT "",
destination_caller
varchar(765) NULL COMMENT "",
recording_url
varchar(765) NULL COMMENT "",
ext_no_answer
array<varchar(255)> NULL COMMENT "" ) how to change data` json NULL COMMENT "" --> String
r
To change the
data
column from
JSON
to
STRING
(which in StarRocks is typically an alias for a long
VARCHAR
), you have two options depending on whether your table already contains data. Option 1: If you are creating a new table Simply replace the
JSON
keyword with
STRING
or
VARCHAR(length)
. In StarRocks,
STRING
is a shorthand for
VARCHAR(65533)
.
Copy code
sql
CREATE TABLE `logcall_crm_tbl` (
  ...
  `data` STRING NULL COMMENT "",  -- Changed from JSON to STRING
  ...
);
Option 2: If the table already exists (with data) StarRocks does not currently support direct type conversion from
JSON
to
STRING
using a single
ALTER TABLE MODIFY COLUMN
statement. You must use the "Add-Update-Drop-Rename" workflow: 1. Add a new temporary column:
Copy code
sql
    ALTER TABLE logcall_crm_tbl ADD COLUMN data_new STRING;
2. Copy and convert the data: You can use the
CAST
function to convert JSON objects into their string representation.
Copy code
sql
    UPDATE logcall_crm_tbl SET data_new = CAST(data AS STRING);
3. Drop the old JSON column:
Copy code
sql
    ALTER TABLE logcall_crm_tbl DROP COLUMN data;
4. Rename the new column:
Copy code
sql
    -- Supported in v3.3.2+
    ALTER TABLE logcall_crm_tbl RENAME COLUMN data_new TO data;
Important Notes: * Storage Size:
STRING
/
VARCHAR
has a maximum length of 65,533 bytes. If your JSON objects are extremely large (MBs), you may need to use the
JSON
type (which supports up to 16MB) or handle them differently. * Querying: Once converted to
STRING
, you can no longer use the arrow operator (
data->'$.id'
). You will have to use functions like
get_json_string(data, '$.id')
. * Wait for completion: After running
ALTER TABLE
(ADD/DROP/RENAME), you can check the status of the background task with:
Copy code
sql
    SHOW ALTER TABLE COLUMN;
References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • sql-reference/data-types/semi_structured/JSON.md • sql-reference/sql-functions/json-functions/get_json_string.md References • using_starrocks/Flat_json.md • sql-reference/data-types/semi_structured/JSON.md
l
@Rocky does data is keyword? how to used this as colume name
r
No,
DATA
is not a reserved keyword
in StarRocks. You can use it as a column name directly without issues. However, it is a common word in SQL, so it is often safer to use backticks (`` ` ``) to avoid any potential ambiguity or conflict with future updates. How to use it as a column name: In your
CREATE TABLE
statement, you can define it like this:
Copy code
sql
CREATE TABLE `logcall_crm_tbl` (
  ...
  `data` STRING NULL COMMENT "",  -- Using backticks is recommended
  ...
);
Key Rules for Column Names: * Backticks: Always use backticks (``
column_name
``) if you want to be 100% safe, especially for names like
data
,
order
,
key
, or
time
. * Characters: Names can include letters (a-z, A-Z), digits (0-9), and underscores (
_
). * Case Sensitivity: Column names are not case-sensitive in StarRocks. Reserved vs. Non-reserved Keywords * Reserved Keywords: Examples include
SELECT
,
TABLE
,
JOIN
,
PRIMARY
. These must be enclosed in backticks if used as names. * Non-reserved Keywords: Examples include
DATA
,
DB
,
STATS
. These can be used without backticks, but using them is still a best practice for clarity. References • sql-reference/sql-statements/keywords.md • sql-reference/System_limit.md References • sql-reference/sql-statements/keywords.md • sql-reference/System_limit.md References • faq/Sql_faq.md • sql-reference/sql-statements/table_bucket_part_index/SELECT.md • sql-reference/sql-statements/keywords.md • administration/management/FE_configuration.md • sql-reference/System_limit.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • release_notes/release-3.3.md
l
@Rocky i saw SR task insert data into masteralized view, if source table is sink by kafka, and have many update, does this effect db insert overwrite
debt_collection_cus_action_hist_raw
SELECT
knowlegedb
.
a
.
ID
,
knowlegedb
.
a
.
SYS_RUN_DATE
,
r
Running an
INSERT OVERWRITE
task while the source table is actively being updated via Kafka (Routine Load) does affect the database, primarily in terms of resource consumption. Here is a breakdown of how it affects the system and how to manage it: 1. Resource Consumption (CPU, Memory, I/O) * Source Table Overhead: If your source table uses the Primary Key model (common for Kafka streams with updates), reading from it during a large
SELECT
requires StarRocks to resolve the latest versions of rows. This consumes extra CPU and memory. * Write Overhead:
INSERT OVERWRITE
writes a completely new version of the data to disk before swapping it. This creates significant disk I/O and triggers background compaction tasks on the target table to merge these new data versions. * Network/Concurrency: If both Kafka ingestion and the
INSERT OVERWRITE
task are running at high frequency, they will compete for the same FE/BE resources, potentially slowing down both the ingestion speed and the query performance. 2. Why
INSERT OVERWRITE
is Safe (Atomicity)
The good news is that
INSERT OVERWRITE
is atomic. * StarRocks creates a temporary partition/table for the new data. * The old data remains available for queries while the task is running. * Only when the task succeeds does StarRocks "swap" the old data with the new. * Result: Your users will never see "partial" data or an empty table during the process. 3. Impact of "Many Updates" If the source table has "many updates," it means the background compaction on that source table is already working hard. Adding a heavy
INSERT OVERWRITE
query adds more pressure. If the cluster is already near 80-90% CPU usage, this task could cause query latency spikes. ────────── Best Practices & Recommendations A. Use Asynchronous Materialized Views (Recommended) Instead of a manual
INSERT OVERWRITE
task, use an Asynchronous Materialized View. * Why: Async MVs can perform incremental refreshes (at the partition level). If only one day of data changed in your Kafka source, StarRocks only refreshes that one partition in the MV, rather than overwriting the whole table. * Syntax:
Copy code
sql
    CREATE MATERIALIZED VIEW my_mv
    REFRESH ASYNC [EVERY INTERVAL]
    AS SELECT ... FROM source_table;
B. Partition-Level Overwrite If you must use
INSERT OVERWRITE
, avoid overwriting the entire table if it is large. If your table is partitioned by date, overwrite only the affected partitions:
Copy code
sql
INSERT OVERWRITE `debt_collection_cus_action_hist_raw` 
PARTITION (p20231027)
SELECT ... FROM ... WHERE date = '2023-10-27';
C. Control Frequency Do not run
INSERT OVERWRITE
too frequently (e.g., every minute). This creates too many data versions, which will eventually slow down the database. For near-real-time needs, the Materialized View approach is much more efficient. D. Check Task Status You can monitor the performance and status of your task using:
Copy code
sql
SELECT * FROM information_schema.task_runs 
WHERE task_name = 'your_task_name' 
ORDER BY create_time DESC;
References • loading/InsertInto.md • using_starrocks/async_mv/Materialized_view.md • faq/loading/Routine_load_faq.md References • faq/loading/Routine_load_faq.md • loading/RoutineLoad.md