http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I have a varchar column written by ...
# questions-and-troubleshooting
m
@Rocky I have a varchar column written by hudi to bucket path with hive as metastore, I have connected external catalog to starrocks 4.0.8 and running select on a table with big column value(could be more than 3MB). I'm getting error reason: Cancelled: Cancelled by pipeline engine, reason: Internal error: going to read out-of-bounds data, offset=1048941,count=1365,size=1048941: file = gs, how can I able to read data of this table(with very big string columns) from starrocks ?
r
In StarRocks (including version 4.0.8), there is a system-wide limit on the maximum length of a single 
VARCHAR
or 
STRING
value, which is 1,048,576 bytes (exactly 1 MB). If you are querying a Hudi table where a column value exceeds 3 MB, you are likely encountering one of the following behaviors based on StarRocks' handling of external catalogs: 1. NULL Values: For Hive and Hudi catalogs, StarRocks typically processes any string value that exceeds the 1 MB limit as a NULL value to prevent the vectorized engine from consuming excessive memory or crashing. 2. Query Errors: In some scenarios, depending on the specific reader implementation, the query may fail with an error like 
Vectorized column ... exceed the limit
. Key Technical Details * Hard Limit: The 1,048,576-byte limit is deep-seated in StarRocks' vectorized execution engine, which uses fixed-size offsets for column data to ensure high performance. While some older documentation mentions a 65,535-byte limit, modern versions (2.1+) support up to 1 MB. * External Tables: The StarRocks documentation explicitly states that for Hive/Hudi external data sources, "The maximum length of a Hive string column that can be queried is 1 MB. If a string column exceeds 1 MB, it will be processed as a null column." Potential Workarounds Currently, there is no official session variable or FE/BE configuration to increase this limit to 3 MB or beyond. However, you can try the following: * Truncation at Source: If you only need a portion of the data, you can create a view in your source system (Hive/Hudi) that uses 
SUBSTRING
to truncate the field to under 1 MB. * JSON Type: If the data is JSON formatted, StarRocks' 
JSON
type also has size limits, but for external scans, it is often still bound by the same underlying binary/string length limits. * Split the Column: If possible, split the large string into multiple smaller columns in the Hudi table and concatenate them in StarRocks using 
CONCAT()
(though the final result of the concatenation will still be subject to the 1 MB limit in the result set). If your use case strictly requires processing single cells larger than 1 MB, you may need to reach out to the StarRocks community on Slack or file a GitHub issue, as this is a known architectural limit of the current vectorized engines designed for performance stability and performance. References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • sql-reference/data-types/string-type/VARCHAR.md • faq/table_design_faq.md • faq/Sql_faq.md
In StarRocks, the 
VARCHAR
and 
STRING
data types have a fundamental architectural limit of 1,048,576 bytes (1 MB) per cell within the vectorized execution engine. The error you are seeing—`Internal error: going to read out-of-bounds data, offset=1048941`—is occurring because the Hudi reader has encountered a field larger than this limit. The offset 
1,048,941
is just slightly over the 1,048,576-byte boundary, triggering a safety check (or a failure to allocate a large enough buffer) in the StarRocks BE (Backend). Why this is happening * Vectorized Engine Limit: StarRocks uses a highly optimized memory layout for strings that uses fixed-width offsets. To maintain this performance, individual string values are capped at 1 MB. * Hudi vs. Hive Behavior: While StarRocks' Hive catalog is often configured to "null out" values that exceed 1 MB to prevent crashes, the Hudi reader in version 4.0.8 appears to be hitting a hard boundary check before it can perform that graceful degradation. Potential Workarounds 1. Avoid selecting the large column If you do not strictly need the data from that specific 3MB+ column for your analysis, the simplest fix is to explicitly list only the columns you need in your 
SELECT
statement, excluding the problematic field. 2. Truncate at the source (Hudi/Hive View) StarRocks must read the entire data cell from the Parquet file before it can apply SQL functions like 
SUBSTR()
. Therefore, trying to truncate the data inside StarRocks often won't prevent the error. * Workaround: Create a view in your Hadoop environment (Hive/Spark) that uses 
substring(column, 1, 1000000)
to ensure no value exceeds 1 MB, then query that view from StarRocks. 3. Use the JSON Type (If applicable) If your large column contains JSON data, StarRocks' native JSON type supports values up to 16 MB. * However, because this is an external Hudi catalog, StarRocks automatically maps Hudi strings to 
VARCHAR
. You cannot easily "re-type" an external column to JSON without changing the source schema or using a complex 
CAST
(which may still hit the 1MB read limit). 4. Check for Bug Fixes in 4.0.9+ The error message you're seeing is a known point of friction in the 4.0.x branch. Version 4.0.9 and later introduced some improvements to string metadata handling (
max_unknown_string_meta_length
). While this may not increase the 1MB limit, it may improve the stability of queries encountering large strings. Summary There is currently no configuration or session variable to increase the 1MB 
VARCHAR
limit to 3MB in StarRocks 4.0.8. If your use case requires processing single cells larger than 1MB, it is highly recommended to: 1. Truncate the data before it reaches StarRocks. 2. Split the large string into multiple columns at the source. 3. Report this as a bug on the StarRocks GitHub specifically as a "Hudi reader out-of-bounds error" so that the team can implement the "graceful NULL" behavior seen in the Hive catalog. References • quick_start/hudi.md • data_source/External_table.md • data_source/catalog/hudi_catalog.md • data_source/feature-support-data-lake-analytics.md
m
@Rocky if data already written to varchar datatype, while reading in starrocks if I cast the column to variant/json, will it work? without throwing out-of-bounds exception?
r
No, casting to 
JSON
or 
VARIANT
in your 
SELECT
statement will not resolve this issue. The "out-of-bounds" error occurs at the storage scanner layer, which happens before the 
CAST
function is ever executed. Here is the technical breakdown of why this happens and what you can do: 1. Why 
CAST
fails * Read-Phase Mapping: When you query a Hudi catalog, StarRocks first maps the Hudi metadata (where your column is a 
string
) to its internal 
VARCHAR
type. * The 1MB Limit: StarRocks' vectorized engine has a hard limit of 1,048,576 bytes (1 MB) for a single cell in a 
VARCHAR
or 
STRING
column. * The Error: During the scan, the Hudi reader attempts to load the 3MB value into a memory buffer allocated for a standard string. Because the value (offset ~1.04MB) exceeds the 1MB safety boundary, the low-level scanner throws the 
Internal error: going to read out-of-bounds data
and cancels the pipeline. * Execution Order: Since the data cannot even be read into memory, the query engine never reaches the "Execution" phase where your 
CAST(... AS JSON)
would process the data. 2. Is 
VARIANT
an option? In StarRocks 4.0.8, the 
VARIANT
type is only supported for the Iceberg Catalog (and requires v4.1+ for broader features). It is not currently supported for the Hudi catalog or native StarRocks tables. Even if it were, the automatic schema mapping for Hudi would still treat the source column as a 
string
, hitting the same 1MB limit. 3. How to read the data Because the limit is in the scanner, you must ensure the scanner never sees a value larger than 1MB. * Truncate at the Source (Recommended): The most reliable way is to create a view in your source system (Spark or Hive) that truncates the column:
Copy code
sql
    -- In Hive/Spark
    CREATE VIEW hudi_truncated_view AS
    SELECT col1, col2, substring(big_varchar_col, 1, 1048576) as big_varchar_col
    FROM your_hudi_table;
Then, query this view from StarRocks. * Use the Hive Catalog: If your Hudi table is compatible with the Hive reader (e.g., a COW table or a Read-Optimized view of a MOR table), try connecting to it via a Hive Catalog instead of a Hudi Catalog. The Hive reader in StarRocks is often more "graceful"—it typically converts strings over 1MB to 
NULL
instead of crashing the query. * Exclude the Column: If you don't need that specific column for your current analysis, avoid using 
SELECT *
. Explicitly select only the columns that are within the size limits. Summary The 1MB limit is an architectural constraint of the StarRocks 
VARCHAR
type designed to prevent memory instability. To process values larger than this, the data must be transformed or truncated before StarRocks attempts to read the Parquet files into its vectorized blocks. References • data_source/External_table.md • quick_start/hudi.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/hudi_catalog.md
2 Views
Open in Slack
PreviousNext
Powered by