<@U0A71G31CDV> i am trying to partition table usin...
# questions-and-troubleshooting
a
@Rocky i am trying to partition table using custom SQL udf global function and getting error
Copy code
SQL Error [1064] [42000]: java.lang.NullPointerException: Cannot invoke "com.starrocks.catalog.UserIdentity.isEphemeral()" because "userIdentity" is null
1
r
The error you are seeing occurs because StarRocks does not support custom SQL UDFs or global functions within the
PARTITION BY
clause of a
CREATE TABLE
statement. Why this happens 1. Restricted Functions: StarRocks expression partitioning only supports a specific list of built-in functions (such as
date_trunc
,
time_slice
,
cast
, and certain time-based math functions). Custom UDFs are not part of the allowed list for defining table storage structure. 2. The Error Message: The
NullPointerException
with
UserIdentity.isEphemeral()
is a bug in the StarRocks SQL analyzer. When it encounters a function it doesn't recognize as a built-in partitioning function, it attempts to resolve it as a general function. This resolution process incorrectly looks for user session context (identity) which isn't fully available during the structural DDL analysis of a table's partition schema. Supported Partitioning Functions To use expression partitioning, you must use one of the supported built-in functions. Common ones include: *
date_trunc
*
time_slice
*
from_unixtime
*
str2date
*
cast
(often used to convert types before applying time functions) Recommended Workarounds * Use Built-in Functions: If your UDF performs common operations like truncating a date or calculating a window, try to rewrite it using the supported functions listed in the Expression Partitioning documentation. * Pre-process Data: If the partitioning logic is highly complex and requires a custom UDF, you should calculate that value at the application level before loading the data, or use an ETL process to create a physical column that StarRocks can partition by directly. * Generated Columns (Alternative): Note that StarRocks currently does not support partitioning by generated columns; partitioning must be based on base columns or supported expressions. If you believe your specific UDF should be supported or want to report the NPE bug, I recommend reaching out on the StarRocks Slack or opening an issue on GitHub. Referencesfaq/Sql_faq.mdsql-reference/sql-functions/JAVA_UDF.mdsql-reference/sql-functions/Python_UDF.mdsql-reference/sql-statements/Function/CREATE_FUNCTION.mdtable_design/data_distribution/expression_partitioning.md
The error you are seeing (
NullPointerException
on
UserIdentity.isEphemeral()
) typically occurs when StarRocks attempts to evaluate a function or expression that requires a user session context in a background process or metadata validation step where that context is missing. In StarRocks, Custom SQL UDFs (including Global UDFs) are currently not supported in partition expressions. Why this is happening: 1. Limited Function Support: StarRocks' "Expression Partitioning" and "Range Partitioning" only support a specific subset of built-in functions (primarily time functions like
date_trunc
,
time_slice
,
from_unixtime
, etc.). 2. Context Missing: When you use a custom SQL UDF in a
PARTITION BY
clause, the StarRocks Frontend (FE) tries to resolve and evaluate that function during the table creation or data loading phase. Because custom UDFs require permission checks and ownership validation (which uses
UserIdentity
), and the partitioning engine often runs these checks in a simplified context, it fails with a
NullPointerException
because it can't find the "user" who is supposed to be running the function. Recommended Solutions: 1. Use Supported Built-in Functions If your UDF performs common time or string manipulations, check if a built-in StarRocks function can replace it. Supported functions for expression partitioning include: *
date_trunc
,
time_slice
*
date_add
,
date_sub
,
datediff
*
cast
,
from_unixtime
,
str2date
2. Transformation at Load Time (ETL) If you need complex logic that only your UDF provides, the standard practice is to: * Add a dedicated column to your table schema (e.g.,
partition_key_col
). * Apply the UDF logic during the data ingestion process (e.g., in the
COLUMNS
clause of a Routine Load, Stream Load, or an
INSERT INTO ... SELECT
statement). * Partition the table by this new column directly. Example: Instead of:
Copy code
sql
-- This will fail
CREATE TABLE my_table (
    id INT,
    raw_time STRING
) PARTITION BY my_custom_udf(raw_time) ...
Use:
Copy code
sql
CREATE TABLE my_table (
    id INT,
    raw_time STRING,
    processed_date DATE  -- Column to hold UDF result
) PARTITION BY (processed_date) ...

-- Then during load:
INSERT INTO my_table
SELECT id, raw_time, my_custom_udf(raw_time) FROM source_data;
Summary The
NullPointerException
you encountered is essentially a bug in how StarRocks reports the lack of support for UDFs in partitioning. You should avoid using UDFs in the
PARTITION BY
clause and instead use built-in functions or pre-process the data into a dedicated partitioning column. Referencestable_design/data_distribution/expression_partitioning.mdtable_design/data_distribution/feature-support-data-distribution.md