Lý Thành
05/29/2026, 2:44 AMRocky
05/29/2026, 2:44 AM\1). RE2 is designed for linear-time matching, and backreferences are excluded because they can lead to exponential-time complexity.
To achieve the same logic (checking if all digits in the string are the same), you can use standard string functions instead of a regular expression.
Recommended Fix
Replace the NOT REGEXP '^([0-9])\1+$' line with a comparison using REPEAT and LEFT. This logic checks if the string is equal to its first character repeated for its entire length:
sql
CASE
WHEN LENGTH(TRIM(ID_CARD_NUMBER_OTHER)) IN (9, 12)
AND TRIM(ID_CARD_NUMBER_OTHER) REGEXP '^[0-9]+$'
AND TRIM(ID_CARD_NUMBER_OTHER) NOT IN ('123456789', '987654321')
-- Check that not all digits are identical (e.g., '111111111')
AND TRIM(ID_CARD_NUMBER_OTHER) != REPEAT(LEFT(TRIM(ID_CARD_NUMBER_OTHER), 1), LENGTH(TRIM(ID_CARD_NUMBER_OTHER)))
THEN TRIM(ID_CARD_NUMBER_OTHER)
ELSE NULL
END AS ID_CARD_NUMBER_OTHER
Alternative Approach (Using REPLACE)
Another way to ensure the string doesn't consist of only one unique digit is to replace all occurrences of the first character with an empty string and check if anything remains:
sql
AND LENGTH(REPLACE(TRIM(ID_CARD_NUMBER_OTHER), LEFT(TRIM(ID_CARD_NUMBER_OTHER), 1), '')) > 0
Why this happened:
* Engine: StarRocks uses the RE2 engine for REGEXP and RLIKE.
* Limitation: RE2 does not support backreferences, lookahead/lookbehind assertions, or other features that require backtracking.
* Performance: While slightly more restrictive, this ensures your queries remain highly performant even with complex data. millions of rows/second even with complex patterns.
References
• sql-reference/sql-functions/string-functions/replace.md
• sql-reference/sql-functions/string-functions/lpad.md
• sql-reference/sql-functions/string-functions/repeat.md
• sql-reference/sql-functions/string-functions/strleft.md
• sql-reference/sql-functions/like-predicate-functions/regexp_replace.md
• sql-reference/sql-functions/string-functions/left.md
• sql-reference/sql-functions/string-functions/rpad.md
• sql-reference/sql-functions/array-functions/array_repeat.mdLý Thành
05/29/2026, 3:10 AMRocky
05/29/2026, 3:11 AM>) and ensured the syntax aligns with StarRocks' SQL parser.
Key Adjustments:
1. Operator Fixes: Replaced > with >.
2. Join Conditions: While StarRocks supports OR in JOIN conditions, note that this can be computationally expensive on very large datasets compared to a UNION of two separate joins. However, it is syntactically valid.
3. Window Functions: The ROW_NUMBER() logic is fully supported in StarRocks.
```sql
WITH
-- 1 LOS_CIF has more than 1 ID_CARD_NUMBER -> take the latest one
source_dup_id_card AS (
SELECT LOS_CIF_NO
FROM source_data_add_master_id_20260430_loan_card
GROUP BY LOS_CIF_NO
HAVING COUNT(DISTINCT ID_CARD_NUMBER) > 1
),
add_MASTER_ID_CARD_dup_id_card AS (
SELECT ID_CARD_NUMBER AS MASTER_ID_CARD, CONTRACT_NO
FROM (
SELECT
ID_CARD_NUMBER,
CONTRACT_NO,
ROW_NUMBER() OVER (PARTITION BY a.LOS_CIF_NO ORDER BY status_created_date DESC) AS RN
FROM source_data_add_master_id_20260430_loan_card a
INNER JOIN source_dup_id_card b ON a.LOS_CIF_NO = b.LOS_CIF_NO
) t
WHERE RN = 1
),
-- 1 ID_CARD_NUMBER has more than 1 LOS_CIF -> take the latest one
source_dup_los_cif AS (
SELECT ID_CARD_NUMBER
FROM source_data_add_master_id_20260430_loan_card
GROUP BY ID_CARD_NUMBER
HAVING COUNT(DISTINCT LOS_CIF_NO) > 1
),
add_MASTER_ID_CARD_dup_los_cif AS (
SELECT ID_CARD_NUMBER AS MASTER_ID_CARD, CONTRACT_NO
FROM (
SELECT
a.ID_CARD_NUMBER,
CONTRACT_NO,
ROW_NUMBER() OVER (PARTITION BY a.LOS_CIF_NO ORDER BY status_created_date DESC) AS RN
FROM source_data_add_master_id_20260430_loan_card a
INNER JOIN source_dup_los_cif b ON a.ID_CARD_NUMBER = b.ID_CARD_NUMBER
) t
WHERE RN = 1
),
-- Cross-match via ID_CARD_NUMBER_OTHER to find the master card
add_master_id_card AS (
SELECT
t1.LOS_CIF_NO,
t1.CONTRACT_NO,
t1.ID_CARD_NUMBER,
t1.ID_CARD_NUMBER_OTHER,
t1.STATUS_CREATED_DATE,
t1.CUSTOMER_NAME,
t2.LOS_CIF_NO AS LOS_CIF_NO_s2,
t2.CONTRACT_NO AS CONTRACT_NO_s2,
t2.ID_CARD_NUMBER AS ID_CARD_NUMBER_s2,
t2.ID_CARD_NUMBER_OTHER AS ID_CARD_NUMBER_OTHER_s2,
t2.STATUS_CREATED_DATE AS STATUS_CREATED_DATE_s2,
CASE
WHEN t1.STATUS_CREATED_DATE > t2.STATUS_CREATED_DATE
THEN t1.ID_CARD_NUMBER
ELSE t2.ID_CARD_NUMBER
END AS MASTER_ID_CARD
FROM source_data_add_master_id_20260430_loan_card t1
INNER JOIN source_data_add_master_id_20260430_loan_card t2
ON t1.ID_CARD_NUMBER_OTHER = t2.ID_CARD_NUMBER
OR t2.ID_CARD_NUMBER_OTHER = t1.ID_CARD_NUMBER
),
-- Deduplicate round 1: keep longest MASTER_ID_CARD per LOS_CIF
add_master_id_card_remove_dup1 AS (
SELECT *
FROM (
SELECT
a.*,
ROW_NUMBER() OVER (PARTITION BY a.LOS_CIF_NO ORDER BY LENGTH(MASTER_ID_CARD) DESC) AS RN
FROM add_master_id_card a
) t
WHERE RN = 1
),
-- Second pass: re-cross-match on the deduplicated set
add_master_id_card_2 AS (
SELECT
t1.LOS_CIF_NO,
t1.CONTRACT_NO,
t1.ID_CARD_NUMBER,
t1.ID_CARD_NUMBER_OTHER,
t1.STATUS_CREATED_DATE,
t1.CUSTOMER_NAME,
t2.LOS_CIF_NO AS LOS_CIF_NO_s2,
t2.CONTRACT_NO AS CONTRACT_NO_s2,
t2.ID_CARD_NUMBER AS ID_CARD_NUMBER_s2,
t2.ID_CARD_NUMBER_OTHER AS ID_CARD_NUMBER_OTHER_s2,
t2.STATUS_CREATED_DATE AS STATUS_CREATED_DATE_s2,
CASE
WHEN LENGTH(t1.MASTER_ID_CARD) > LENGTH(t2.MASTER_ID_CARD)
THEN t1.MASTER_ID_CARD
ELSE t2.MASTER_ID_CARD
END AS MASTER_ID_CARD
FROM add_master_id_card_remove_dup1 t1Rocky
05/29/2026, 3:11 AMOR conditions (e.g., in add_master_id_card). If your dataset is very large (hundreds of millions of rows), StarRocks might struggle to optimize this specific join type. If you experience slowness, consider breaking the OR join into two separate INNER JOIN blocks and using UNION ALL to combine them.Lý Thành
05/29/2026, 3:13 AMOR join into two separate INNER JOIN blocks and using UNION ALL to combine them.Rocky
05/29/2026, 3:13 AMOR joins into separate INNER JOIN blocks and combine them with UNION ALL. This allows the StarRocks optimizer to use more efficient hash join strategies for each individual join.
Since your downstream logic uses ROW_NUMBER() ... WHERE RN = 1, the extra rows generated by UNION ALL (in cases where both OR conditions are met) will be automatically deduplicated.
Here is the rewritten SQL:
```sql
WITH
-- 1 LOS_CIF has more than 1 ID_CARD_NUMBER -> take the latest one
source_dup_id_card AS (
SELECT LOS_CIF_NO
FROM source_data_add_master_id_20260430_loan_card
GROUP BY LOS_CIF_NO
HAVING COUNT(DISTINCT ID_CARD_NUMBER) > 1
),
add_MASTER_ID_CARD_dup_id_card AS (
SELECT ID_CARD_NUMBER AS MASTER_ID_CARD, CONTRACT_NO
FROM (
SELECT
ID_CARD_NUMBER,
CONTRACT_NO,
ROW_NUMBER() OVER (PARTITION BY a.LOS_CIF_NO ORDER BY status_created_date DESC) AS RN
FROM source_data_add_master_id_20260430_loan_card a
INNER JOIN source_dup_id_card b ON a.LOS_CIF_NO = b.LOS_CIF_NO
) t
WHERE RN = 1
),
-- 1 ID_CARD_NUMBER has more than 1 LOS_CIF -> take the latest one
source_dup_los_cif AS (
SELECT ID_CARD_NUMBER
FROM source_data_add_master_id_20260430_loan_card
GROUP BY ID_CARD_NUMBER
HAVING COUNT(DISTINCT LOS_CIF_NO) > 1
),
add_MASTER_ID_CARD_dup_los_cif AS (
SELECT ID_CARD_NUMBER AS MASTER_ID_CARD, CONTRACT_NO
FROM (
SELECT
a.ID_CARD_NUMBER,
CONTRACT_NO,
ROW_NUMBER() OVER (PARTITION BY a.LOS_CIF_NO ORDER BY status_created_date DESC) AS RN
FROM source_data_add_master_id_20260430_loan_card a
INNER JOIN source_dup_los_cif b ON a.ID_CARD_NUMBER = b.ID_CARD_NUMBER
) t
WHERE RN = 1
),
-- REWRITTEN: Cross-match via UNION ALL instead of OR
add_master_id_card AS (
SELECT
t1.LOS_CIF_NO, t1.CONTRACT_NO, t1.ID_CARD_NUMBER, t1.ID_CARD_NUMBER_OTHER, t1.STATUS_CREATED_DATE, t1.CUSTOMER_NAME,
t2.LOS_CIF_NO AS LOS_CIF_NO_s2, t2.CONTRACT_NO AS CONTRACT_NO_s2, t2.ID_CARD_NUMBER AS ID_CARD_NUMBER_s2,
t2.ID_CARD_NUMBER_OTHER AS ID_CARD_NUMBER_OTHER_s2, t2.STATUS_CREATED_DATE AS STATUS_CREATED_DATE_s2,
CASE WHEN t1.STATUS_CREATED_DATE > t2.STATUS_CREATED_DATE THEN t1.ID_CARD_NUMBER ELSE t2.ID_CARD_NUMBER END AS MASTER_ID_CARD
FROM source_data_add_master_id_20260430_loan_card t1
INNER JOIN source_data_add_master_id_20260430_loan_card t2 ON t1.ID_CARD_NUMBER_OTHER = t2.ID_CARD_NUMBER
UNION ALL
SELECT
t1.LOS_CIF_NO, t1.CONTRACT_NO, t1.ID_CARD_NUMBER, t1.ID_CARD_NUMBER_OTHER, t1.STATUS_CREATED_DATE, t1.CUSTOMER_NAME,
t2.LOS_CIF_NO AS LOS_CIF_NO_s2, t2.CONTRACT_NO AS CONTRACT_NO_s2, t2.ID_CARD_NUMBER AS ID_CARD_NUMBER_s2,
t2.ID_CARD_NUMBER_OTHER AS ID_CARD_NUMBER_OTHER_s2, t2.STATUS_CREATED_DATE AS STATUS_CREATED_DATE_s2,
CASE WHEN t1.STATUS_CREATED_DATE > t2.STATUS_CREATED_DATE THEN t1.ID_CARD_NUMBER ELSE t2.ID_CARD_NUMBER END AS MASTER_ID_CARD
FROM source_data_add_master_id_20260430_loan_card t1
INNER JOIN source_data_add_master_id_20260430_loan_card t2 ON t2.ID_CARD_NUMBER_OTHER = t1.ID_CARD_NUMBER
),
-- Deduplicate round 1
add_master_id_card_remove_dup1 AS (
SELECT * FROM (
SELECT a.*, ROW_NUMBER() OVER (PARTITION BY a.LOS_CIF_NO ORDER BY LENGTH(MASTER_ID_CARD) DESC) AS RN
FROM add_master_id_card a
) t WHERE RN = 1
),
-- REWRITTEN: Second pass via UNION ALL instead of OR
add_master_id_card_2 AS (
SELECT
t1.LOS_CIF_NO, t1.CONTRACT_NO, t1.ID_CARD_NUMBER, t1.ID_CARD_NUMBER_OTHER, t1.STATUS_CREATED_DATE, t1.CUSTOMER_NAME,
t2.LOS_CIF_NO AS LOS_CIF_NO_s2, t2.CONTRACT_NO AS CONTRACT_NO_s2, t2.ID_CARD_NUMBER AS ID_CARD_NUMBER_s2,
t2.ID_CARD_NUMBER_OTHER AS ID_CARD_NUMBER_OTHER_s2, t2.STATUS_CREATED_DATE AS STATUS_CREATED_DATE_s2,Rocky
05/29/2026, 3:13 AMA.col = B.col). By using UNION ALL, each sub-query performs a standard hash join.
* Parallelism: StarRocks can execute the different branches of a UNION ALL in parallel across the cluster.
* Avoids Nested Loops: Joins with OR often default to less efficient join types because the engine cannot easily build a single hash table for multiple independent columns from both sides of the OR.
References
• knowledge/usage/best-practice.md
• faq/Sql_faq.md
• sql-reference/sql-statements/table_bucket_part_index/SELECT.md
• using_starrocks/accelerate-cardinality-preserving-joins.mdLý Thành
05/29/2026, 7:11 AMRocky
05/29/2026, 7:11 AMTRANSLATE function (available from v3.2 onwards) and the REGEXP_REPLACE function.
Since the || operator in StarRocks defaults to a logical OR (unless the pipes_as_concat session variable is enabled), it is safer and more standard to use the CONCAT() function or provide the strings as single long literals.
Rewritten SQL
sql
SELECT a.*,
UPPER(
REGEXP_REPLACE(
TRANSLATE(
UPPER(CUSTOMER_NAME_REMOVE_COMBINING),
-- From string (all accented characters)
CONCAT(
'ÀÁẠẢÃÂẦẤẬẨẪĂẰẮẶẲẴĀĄÃÆÄÅ',
'ÈÉẸẺẼÊỀẾỆỂỄĔĖĘĚ',
'ÌÍỊỈĨĬĮİ',
'ÒÓỌỎÕÔỒỐỘỔỖƠỜỚỢỞỠŎŐØ',
'ÙÚỤỦŨƯỪỨỰỬỮŬŮŰŲǓ',
'ỲÝỴỶỸ',
'ĐĎ',
'ĆČÇ',
'ÑŃŇŅŊ',
'Ř',
'ŚŠŞ',
'ŹŽŻ'
),
-- To string (non-accented equivalents)
CONCAT(
'AAAAAAAAAAAAAAAAAAAAAAA',
'EEEEEEEEEEEEEEE',
'IIIIIIII',
'OOOOOOOOOOOOOOOOOOOO',
'UUUUUUUUUUUUUUUU',
'YYYYY',
'DD',
'CCC',
'NNNNN',
'R',
'SSS',
'ZZZ'
)
),
'[[:space:]]+', -- Matches one or more whitespace characters
' ' -- Replaces with a single space
)
) AS CUSTOMER_NAME_NO_ACCENT
FROM convert_customer_name_STEP_2 a;
Key Considerations for StarRocks:
1. Version Requirement: The TRANSLATE function was introduced in StarRocks v3.2. If you are on an older version, you would need to nest multiple REPLACE() calls (which is inefficient for this many characters) or use a Custom Function (UDF).
2. Regular Expression Engine: StarRocks uses the RE2 engine. The pattern [[:space:]]+ is fully supported to identify whitespace (including tabs and various space characters) and collapse them into one.
3. String Concatenation: If you prefer using || for concatenation as in your original snippet, you can run SET pipes_as_concat = true; before your query, or simply use CONCAT() as shown above to avoid issues.
4. Character Counts: Ensure the number of characters in the from_string exactly matches the to_string. In the provided logic, there are 104 characters in both sets.
References
• sql-reference/sql-functions/string-functions/translate.md
• sql-reference/sql-functions/string-functions/replace.md
• sql-reference/sql-functions/string-functions/upper.md
• sql-reference/sql-functions/like-predicate-functions/regexp_replace.md
• sql-reference/sql-functions/string-functions/concat.md
• sql-reference/sql-functions/string-functions/trim.md
• sql-reference/sql-statements/TRANSLATE_TRINO.md
• sql-reference/sql-functions/string-functions/ltrim.md
• sql-reference/sql-functions/string-functions/rtrim.md
References
• sql-reference/sql-functions/string-functions/translate.mdRocky
05/29/2026, 7:11 AMLý Thành
05/29/2026, 7:50 AMRocky
05/29/2026, 7:50 AMWITH RECURSIVE / walk logic) or the CYCLE clause. In a distributed OLAP database like StarRocks, graph traversal is typically handled by:
1. Iterative Joining: Performing a fixed number of self-joins (usually 3-4 hops cover 99% of real-world identity links).
2. External Processing: Using Spark/GraphX for complex clustering.
I have rewritten the query below using 3 levels of joins to simulate the recursion (sufficient for most KYC/Master Data use cases) and adjusted the functions for StarRocks compatibility.
Optimized StarRocks SQL
```sql
WITH
-- Step 1: Unpivot IDs. Use UNION (distinct) to reduce rows early.
id_map AS (
SELECT LOS_CIF_NO, ID_CARD_NUMBER AS ID_VALUE FROM MASTER_ID_CARD_ALL_20260430_loan_card_final_v2 WHERE ID_CARD_NUMBER IS NOT NULL
UNION
SELECT LOS_CIF_NO, ID_CARD_NUMBER_OTHER AS ID_VALUE FROM MASTER_ID_CARD_ALL_20260430_loan_card_final_v2 WHERE ID_CARD_NUMBER_OTHER IS NOT NULL
UNION
SELECT LOS_CIF_NO, MASTER_ID_CARD AS ID_VALUE FROM MASTER_ID_CARD_ALL_20260430_loan_card_final_v2 WHERE MASTER_ID_CARD IS NOT NULL
),
-- Step 2: Direct edges between CIFs sharing an ID
edges AS (
SELECT DISTINCT
a.LOS_CIF_NO AS FROM_CIF,
b.LOS_CIF_NO AS TO_CIF
FROM id_map a
JOIN id_map b ON a.ID_VALUE = b.ID_VALUE
WHERE a.LOS_CIF_NO <> b.LOS_CIF_NO
),
-- Step 3: Simulate Recursion (3-hop connection)
-- Since StarRocks lacks RECURSIVE CTE, we join edges to find the 'ultimate' root
hop1 AS (
SELECT FROM_CIF, TO_CIF FROM edges
),
hop2 AS (
SELECT h1.FROM_CIF, COALESCE(h2.TO_CIF, h1.TO_CIF) AS TO_CIF
FROM hop1 h1
LEFT JOIN edges h2 ON h1.TO_CIF = h2.FROM_CIF
),
hop3 AS (
SELECT h2.FROM_CIF, COALESCE(h3.TO_CIF, h2.TO_CIF) AS TO_CIF
FROM hop2 h2
LEFT JOIN edges h3 ON h2.TO_CIF = h3.FROM_CIF
),
-- Combine all paths and find the minimum CIF to represent the cluster
all_connections AS (
SELECT LOS_CIF_NO AS FROM_CIF, LOS_CIF_NO AS TO_CIF FROM id_map
UNION ALL
SELECT FROM_CIF, TO_CIF FROM hop3
),
component_map AS (
SELECT
FROM_CIF AS LOS_CIF_NO,
MIN(TO_CIF) AS LOS_CIF_NO_V2
FROM all_connections
GROUP BY FROM_CIF
),
-- Step 5: Rank Master IDs within the cluster
master_rank AS (
SELECT
cm.LOS_CIF_NO_V2,
b.MASTER_ID_CARD,
COUNT(*) AS CNT_MASTER,
MAX(b.STATUS_CREATED_DATE) AS MAX_STATUS_DATE,
ROW_NUMBER() OVER (
PARTITION BY cm.LOS_CIF_NO_V2
ORDER BY COUNT(*) DESC, MAX(b.STATUS_CREATED_DATE) DESC, b.MASTER_ID_CARD DESC
) AS RN
FROM MASTER_ID_CARD_ALL_20260430_loan_card_final_v2 b
JOIN component_map cm ON b.LOS_CIF_NO = cm.LOS_CIF_NO
WHERE b.MASTER_ID_CARD IS NOT NULL
GROUP BY cm.LOS_CIF_NO_V2, b.MASTER_ID_CARD
),
group_master AS (
SELECT LOS_CIF_NO_V2, MASTER_ID_CARD AS MASTER_ID_CARD_V2
FROM master_rank
WHERE RN = 1
),
data_LOS_CIF_NO_v2 AS (
SELECT
b.*,
IFNULL(cm.LOS_CIF_NO_V2, b.LOS_CIF_NO) AS LOS_CIF_NO_V2,
IFNULL(gm.MASTER_ID_CARD_V2, b.MASTER_ID_CARD) AS MASTER_ID_CARD_V2
FROM MASTER_ID_CARD_ALL_20260430_loan_card_final_v2 b
LEFT JOIN component_map cm ON b.LOS_CIF_NO = cm.LOS_CIF_NO
LEFT JOIN group_master gm ON cm.LOS_CIF_NO_V2 = gm.LOS_CIF_NO_V2
),
-- Step 7: String Normalization (Optimized for StarRocks)
convert_customer_name_STEP_1 AS (
SELECT *,
REGEXP_REPLACE(
TRIM(REPLACE(REPLACE(REPLACE(CUSTOMER_NAME, '\t', ' '), '\n', ' '), '\r', ' ')),
'[[space]]+', ' '
) AS CUSTOMER_NAME_TRIM
FROM data_LOS_CIF_NO_v2
),
customer_name_NORMALIZED AS (
SELECT *,
UPPER(
REGEXP_REPLACE(
TRANSLATE(
UPPER(CUSTOMER_NAME_TRIM),Rocky
05/29/2026, 7:50 AMhop1, hop2, hop3. This joins edges multiple times to flatten the hierarchy. In banking data, it is extremely rare for a customer identity chain to exceed 3 hops (CIF A -> ID 1 -> CIF B -> ID 2 -> CIF C). This is significantly faster in StarRocks than recursion.
2. String Functions:
* TRANSLATE: Used for bulk character replacement. Ensure you are on StarRocks v3.2+.
* `CHR()`/`UNISTR()`: Replaced CHR(9), CHR(10), etc., with their literal escape counterparts (\t, \n, \r) which StarRocks handles more efficiently in REPLACE functions.
* CONCAT: Used CONCAT() instead of || to avoid issues with the pipes_as_concat setting.
3. Performance Tuning:
* UNION ALL vs UNION: In all_connections, I used UNION ALL because the final GROUP BY in component_map will handle the deduplication more efficiently than a distinct sort in the middle of the CTE.
* IFNULL: Replaced NVL with IFNULL (standard StarRocks syntax, though NVL is supported as an alias).
* DISTINCT Reduction: Added a data_name_rank CTE to handle the final sequence numbering, reducing the number of joins on large datasets.
What if I need more than 3 hops?
If you have data where one identity chain spans 10+ CIFs, the hop logic will need more levels. However, for 99% of "Customer 360" or "Single Customer View" projects, 3 hops are the sweet-spot for performance vs performance-spot.Lý Thành
05/29/2026, 8:33 AMRocky
05/29/2026, 8:34 AMsql
CREATE TABLE QTDM.MASTER_ID_CARD_ALL_20260430_LOAN_CARD_FINAL_V2_CONVERT_LOS_CIF_SR (
-- Grouping columns first for better compression and indexing (Key Columns)
LOS_CIF_NO VARCHAR(100),
CONTRACT_NO VARCHAR(500),
APP_CODE VARCHAR(100),
ID_CARD_NUMBER VARCHAR(100),
-- Other identifying columns
CONTRACT_STATUS VARCHAR(50),
ID_CARD_NUMBER_OTHER VARCHAR(100),
STATUS_CREATED_DATE DATETIME, -- Oracle DATE contains time, so DATETIME is the correct match
CUSTOMER_NAME VARCHAR(1000),
TYPE CHAR(16),
MASTER_ID_CARD VARCHAR(100),
LOS_CIF_NO_V2 VARCHAR(100),
MASTER_ID_CARD_V2 VARCHAR(100),
-- Long string columns (normalized data)
-- VARCHAR in StarRocks supports up to 1,048,576 bytes
CUSTOMER_NAME_TRIM VARCHAR(65533),
CUSTOMER_NAME_REMOVE_COMBINING VARCHAR(65533),
CUSTOMER_NAME_NO_ACCENT VARCHAR(65533),
CUSTOMER_NAME_CONVERT VARCHAR(65533),
CUSTOMER_NAME_CONVERT_REMOVE_SPACE VARCHAR(65533),
TEXT_CHECK_STATUS VARCHAR(50),
SPECIAL_CHAR_FOUND VARCHAR(65533),
CUSTOMER_NAME_ASCIISTR VARCHAR(5000),
-- Metrics / Counts
CNT_CONTRACT BIGINT,
CNT_LOS_CIF BIGINT,
CNT_NAME BIGINT,
MASTER_ID_CARD_V3 VARCHAR(200)
)
ENGINE=OLAP
DUPLICATE KEY(LOS_CIF_NO, CONTRACT_NO, APP_CODE, ID_CARD_NUMBER)
COMMENT "Master ID Card Final Normalized Table"
DISTRIBUTED BY HASH(LOS_CIF_NO) BUCKETS 16
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "v2"
);
Key Differences & Mapping Logic:
1. Data Types:
* VARCHAR2 → VARCHAR: StarRocks VARCHAR is length-limited by bytes. I expanded the lengths (e.g., 50 → 100) to account for potential UTF-8 multi-byte characters safely.
* DATE → DATETIME: In Oracle, DATE stores year, month, day, hour, minute, and second. In StarRocks, DATE is only YYYY-MM-DD. Use DATETIME to preserve the time component.
* NUMBER → BIGINT: For columns like CNT_CONTRACT, which are counts, BIGINT is more performant than DECIMAL.
* Large VARCHAR2(32767): StarRocks handles very large strings easily in VARCHAR. I set them to 65533, which is a common safe upper bound for standard columns, though they can go up to 1MB if needed.
2. Table Model (DUPLICATE KEY):
* Oracle is a row-store; StarRocks is a column-store. The DUPLICATE KEY model is ideal here because it allows duplicate rows but sorts the data based on the key columns, making filters on LOS_CIF_NO or CONTRACT_NO extremely fast.
3. Distribution (DISTRIBUTED BY HASH):
* I chose LOS_CIF_NO for the hash distribution. This ensures that all data for the same customer is physically co-located on the same machine, making the JOIN and GROUP BY operations from your previous queries much faster.
4. Storage Parameters:
* Oracle-specific physical attributes like PCTFREE, TABLESPACE, STORAGE, and INITRANS are ignored in StarRocks. Instead, StarRocks uses replication_num (for high availability) and buckets (for parallelism).
Recommendations:
* Buckets: I set BUCKETS 16. If your table size is expected to be over 100GB, you should increase this number (usually 1 bucket per 5-10GB of data per tablet).
* Partitioning: If you plan to keep years of history and query by date often, you can add PARTITION BY RANGE(STATUS_CREATED_DATE) to the DDL.Lý Thành
05/29/2026, 9:28 AMRocky
05/29/2026, 9:29 AMsql
CREATE EXTERNAL CATALOG my_mysql_db
PROPERTIES (
"type" = "jdbc",
"user" = "root",
"password" = "your_password",
"jdbc_uri" = "jdbc:<mysql://192.168.1.1:3306/target_db>",
"driver_url" = "<https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar>",
"driver_class" = "com.mysql.cj.jdbc.Driver"
);
2. Insert Data: Use the INSERT INTO statement referencing the catalog, database, and table.
sql
INSERT INTO my_mysql_db.target_db.external_table
SELECT * FROM internal_starrocks_table;
Key Considerations
* Privileges: You must have the INSERT privilege on the target external table.
* Network Connectivity: Your StarRocks FE and BE (or CN) nodes must have network access to the target database's IP and port.
* Drivers: The JDBC driver .jar file must be accessible via the driver_url. FEs download it upon catalog creation, and BEs download it during the first query/insert.
* Performance: While convenient, sinking data via JDBC is generally slower than StarRocks' native internal storage. It is best used for small to medium data transfers or metadata updates.
For more information, you can check the StarRocks JDBC Catalog documentation.
References
• data_source/catalog/jdbc_catalog.md
• data_source/catalog/unified_catalog.md
• data_source/catalog/hive_catalog.mdLý Thành
06/01/2026, 8:09 AMRocky
06/01/2026, 8:09 AMSTART parameter to anchor the first execution time and the EVERY(INTERVAL 1 DAY) parameter for the frequency.
StarRocks SUBMIT TASK (supported since v3.3 for scheduled tasks) uses a specific interval-based syntax rather than standard Cron expressions.
Modified SQL
Replace the START date with your desired first execution date (e.g., tomorrow at 080000):
sql
SUBMIT TASK losrep_application
SCHEDULE START('2024-05-20 08:00:00') EVERY(INTERVAL 1 DAY)
AS
INSERT INTO DMAPP.dtm_cus_applications_af BY NAME
SELECT * FROM dmapp_link.dtm_cus_applications_af_link;
Key Parameters:
* START('YYYY-MM-DD 08:00:00'): This sets the specific time of day for the first run. All subsequent runs will occur exactly 24 hours (1 day) after this anchor point.
* EVERY(INTERVAL 1 DAY): Defines the recurring frequency.
* BY NAME: Your use of INSERT ... BY NAME is supported in StarRocks and is recommended when the source and target table schemas might have different column orders.
Monitoring your Task:
Once submitted, you can monitor the task's schedule and history using these metadata views:
1. Check the task definition and next trigger time:
sql
SELECT * FROM INFORMATION_SCHEMA.tasks WHERE task_name = 'losrep_application';
2. Check the execution history (status of each run):
sql
SELECT * FROM INFORMATION_SCHEMA.task_runs WHERE task_name = 'losrep_application';
Notes:
* Version requirement: Scheduled intervals for SUBMIT TASK require StarRocks v3.3 or higher.
* Modifying the task: If you need to change the schedule later, you must DROP TASK losrep_application and submit it again, or use ALTER TASK (supported from v4.1+).
References
• sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md
• integrations/airflow.mdLý Thành
06/01/2026, 9:18 AMRocky
06/01/2026, 9:18 AMSELECT list (370) to exactly match the number of columns in the target table (373).
Even if you intend to use "insert by name," the keywords BY NAME must be explicitly included in the statement. In the snippet you provided, they are missing.
Correct Syntax
To insert only the columns that match by name (and fill missing target columns with NULL or their DEFAULT values), you must place BY NAME immediately after the table name:
sql
INSERT INTO dtm_customer_ld_summary BY NAME -- <--- Add this keyword
SELECT *
FROM dmapp_link.dtm_customer_ld_summary_link a
WHERE a.SYS_RUN_DATE = %s;
Why this works:
1. Column Mapping: When BY NAME is present, StarRocks ignores the order and the total count of columns. It looks at the column names returned by your JDBC link (dmapp_link) and matches them to the column names in dtm_customer_ld_summary.
2. Handling Differences:
* If the target table has columns that don't exist in the source, they are filled with DEFAULT or NULL.
* If the source has columns that don't exist in the target, they are typically ignored.
3. Requirements:
* Version: This feature requires StarRocks v3.2.5+ or v3.3.0+.
* JDBC Metadata: Since you are using a JDBC link, StarRocks must be able to retrieve the column names from the external database. If the external driver doesn't provide names, BY NAME will fail. (Most standard JDBC drivers for Oracle/MySQL/Postgres work fine).
Troubleshooting
If you added BY NAME and still get a column count error, it usually means:
* Version too old: Check your version with SELECT asterisk_version();. If you are on an older version (e.g., 2.5 or 3.1), BY NAME is not supported, and you must explicitly list the 370 column names in the INSERT clause.
* Ambiguous Names: If your SELECT statement has duplicate column names (rare with SELECT *), it may cause an error.
Recommendation: If you are on an older version, you must manually align the columns:
sql
INSERT INTO dtm_customer_ld_summary (col1, col2, col3, ...)
SELECT col1, col2, col3, ... FROM dmapp_link...
References
• loading/InsertInto.md
• sql-reference/sql-statements/loading_unloading/INSERT.md
References
• loading/InsertInto.md
• sql-reference/sql-statements/loading_unloading/INSERT.mdLý Thành
06/01/2026, 10:20 AMRocky
06/01/2026, 10:20 AMlast_promised_amount to the type DECIMAL(20, 2), you should use the ALTER TABLE ... MODIFY COLUMN statement.
However, please note that DECIMAL64 only supports up to 18 digits of precision. Since you requested a precision of 20, the column will automatically be stored as a DECIMAL128 internally. In StarRocks SQL, you simply define this as DECIMAL(20, 2).
SQL Command
sql
ALTER TABLE your_table_name
MODIFY COLUMN last_promised_amount DECIMAL(20, 2);
Important Considerations:
1. Internal Types:
* `DECIMAL64`: Precision 1 to 18.
* `DECIMAL128`: Precision 19 to 38.
* Because 20 > 18, StarRocks will treat DECIMAL(20, 2) as a 128-bit decimal.
2. Asynchronous Operation: Modifying a column type is an asynchronous "Schema Change" operation. You can check the progress of the update using:
sql
SHOW ALTER TABLE COLUMN;
3. Renaming: If you also intended to rename a different column to last_promised_amount, you must do this in two steps (requires StarRocks v3.3.2+ for renaming):
sql
-- Step 1: Rename the column
ALTER TABLE your_table_name RENAME COLUMN old_name TO last_promised_amount;
-- Step 2: Modify the type
ALTER TABLE your_table_name MODIFY COLUMN last_promised_amount DECIMAL(20, 2);
4. Restrictions:
* You cannot modify partition columns.
* The conversion must be compatible (e.g., StarRocks will allow converting INT or VARCHAR to DECIMAL, but ensure your existing data fits the (20, 2) scale to avoid data loss or NULLs).
References
• sql-reference/data-types/numeric/DECIMAL.md
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
References
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
• best_practices/query_tuning/schema_tuning.mdLý Thành
06/02/2026, 2:41 AMRocky
06/02/2026, 2:41 AMLý Thành
06/02/2026, 3:28 AMRocky
06/02/2026, 3:28 AMCURRENT_USER trong StarRocks bằng cách sử dụng dấu nháy ngược (backticks - `` ` ``) để bao quanh tên cột.
Trong StarRocks (và MySQL), CURRENT_USER là một từ khóa dự phòng (reserved keyword) dùng cho hàm hệ thống. Tuy nhiên, bạn vẫn có thể đặt tên cột trùng với từ khóa này nếu tuân theo quy tắc sau:
1. Khi tạo bảng (DDL)
Bạn hãy bao quanh tên cột bằng dấu nháy ngược:
sql
CREATE TABLE DTM_LOS_APP_PROCESSING_HIST (
id INT,
`CURRENT_USER` VARCHAR(255), -- Sử dụng dấu nháy ngược ở đây
...
) ...;
2. Khi thực hiện lệnh INSERT/SELECT
Mỗi khi truy vấn hoặc thao tác với cột này, bạn bắt buộc phải dùng dấu nháy ngược, nếu không StarRocks sẽ hiểu lầm đó là hàm hệ thống CURRENT_USER() và gây lỗi:
sql
INSERT INTO DTM_LOS_APP_PROCESSING_HIST (`CURRENT_USER`, column2)
SELECT `CURRENT_USER`, column2
FROM source_table;
Lưu ý quan trọng:
* JDBC Link/External Table: Nếu bạn đang dùng JDBC Link để kéo dữ liệu từ Oracle, tại phần định nghĩa bảng trong StarRocks, bạn cũng dùng dấu nháy ngược cho cột đó.
* Công cụ báo cáo: Nếu bạn dùng JasperReports hay các công cụ BI khác, hãy đảm bảo các câu lệnh SQL mà công cụ đó sinh ra có hỗ trợ dấu nháy ngược cho identifiers (thường cấu hình qua Driver MySQL/StarRocks).
* Khuyên dùng: Mặc dù dùng được, nhưng để tránh việc phải luôn nhớ thêm dấu nháy ngược (dễ gây lỗi syntax khi viết SQL nhanh), các kỹ sư dữ liệu thường ưu tiên đặt tên hơi khác đi một chút (như USER_NAME hoặc CURRENT_USER_VAL). Tuy nhiên, nếu yêu cầu bắt buộc giống Oracle 100%, hãy dùng `` CURRENT_USER ``.
Tóm lại: Bạn không cần đổi thành CURRENT_USER_1, chỉ cần dùng `` CURRENT_USER `` trong mọi câu lệnh SQL.
References
• sql-reference/sql-statements/keywords.md
• faq/Sql_faq.md
References
• faq/Sql_faq.md
• sql-reference/sql-statements/keywords.md
• sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md
• sql-reference/sql-statements/table_bucket_part_index/SELECT.md
• administration/management/FE_configuration.md
• faq/loading/Stream_load_faq.md
• data_source/External_table.md