Hi Team, is there any way I can read string column written by spark in hudi table which has more than 1MB in column size from StarRocks(hive external catalog)? @Rocky getting ERROR 5609 (22000): going to read out-of-bounds data, offset=1048620,count=1234,size=1048620: file = gs://

The error

ERROR 5609 (22000): going to read out-of-bounds data

occurs because StarRocks has a default limit of 1 MB (1,048,576 bytes) for

STRING

and

VARCHAR

data types. When the Parquet reader (used by Hive/Hudi catalogs) encounters a single string value in a page that exceeds this 1 MB buffer, it attempts to read beyond the allocated memory, triggering the "out-of-bounds" error. This is a known limitation when reading data written by Spark, which often allows much larger string values. Solutions and Workarounds 1. Increase the Parquet Column Size Limit (Recommended for v3.2.x+) Starting from StarRocks version 3.2, you can increase the maximum allowed size for a Parquet column string via a session variable. Set this to 16 MB (16,777,216 bytes) or higher to accommodate your data:

sql
-- Set for the current session
SET parquet_max_column_size = 16777216;

-- Or set globally for all new sessions
SET GLOBAL parquet_max_column_size = 16777216;

2. Use the JSON Data Type (Workaround for older versions) If you are on an older version or the variable above does not work, the

JSON

data type in StarRocks natively supports values up to 16 MB. * For External Catalogs: You cannot easily change the mapping of an auto-inferred catalog column. However, you can create a View in StarRocks that casts the problematic column to JSON:

sql
    CREATE VIEW hudi_large_string_view AS
    SELECT CAST(large_string_col AS JSON) as large_string_col, ...
    FROM hudi_catalog.db.table;

* For File External Tables: If you are using

CREATE EXTERNAL TABLE ... ENGINE=file

, manually define the column as

JSON

instead of

STRING

in the DDL. 3. Redefine as VARIANT (v3.2.x+) The

VARIANT

type also supports up to 16 MB. If the column contains semi-structured data, mapping it to

VARIANT

can resolve the buffer issues while providing better performance for nested access. Why this happens In the StarRocks C++ execution engine, the Parquet scanner allocates a fixed-size buffer for string columns to optimize performance. When Spark writes a Hudi table with massive strings (e.g., large JSON blobs or logs in a single cell), it may exceed the default 1 MB scanner buffer. The specific offset and size in your error (

offset=1048620

) are right at the 1,048,576-byte boundary, confirming this limit was hit. Note: If your strings are even larger than 16 MB, StarRocks may still struggle to process them as a single cell, as 16 MB is the current hard limit for semi-structured types (JSON/VARIANT) as well. References • administration/management/BE_configuration.md

@Rocky there is no config like parquet_max_column_size in StarRocks

This error occurs because the StarRocks Parquet scanner has a default limit of 1 MB (1,048,576 bytes) for the memory buffer of a single

STRING

or

VARCHAR

cell. Your data (at offset 1,048,620) is just slightly over this 1 MB boundary, causing the reader to fail with an "out-of-bounds" error. Solution (for StarRocks v3.2.10+ and v3.3.0+) In newer versions of StarRocks, you can increase this limit using a session variable or a BE configuration. 1. Session Variable (Recommended) You can set this for your current session or globally. Set it to 8 MB (8,388,608) or higher depending on your largest string:

sql
-- Set for the current session
SET parquet_max_column_size = 8388608;

-- Or set globally for all new sessions
SET GLOBAL parquet_max_column_size = 8388608;

2. BE Configuration If you want to change it at the cluster level in

be.conf

, add:

conf
parquet_max_column_size = 8388608

(A restart of the BE nodes is required for this to take effect). If the config is missing (Older Versions) If you are on an older version (e.g., v2.5, v3.1.x, or early v3.2.x), the variable

parquet_max_column_size

does not exist because the 1 MB limit was hardcoded. * Upgrade: The permanent fix is to upgrade to at least v3.2.10 or v3.3.0, where this parameter was introduced to handle large strings in Parquet files. * Workaround (JSON Type): If your large string is JSON-formatted, the

JSON

and

VARIANT

data types in StarRocks support up to 16 MB. However, because you are using a Hive/Hudi catalog, StarRocks infers the schema as

STRING

automatically. You might try creating a StarRocks View that explicitly casts the column:

sql
    CREATE VIEW hudi_large_string_view AS
    SELECT CAST(large_col AS JSON) as large_col FROM hudi_catalog.db.table;

Note: This only works if the scanner recognizes the intent to read as JSON before hitting the string buffer limit, which varies by version. Why this happens Spark (and other Hadoop-based writers) allows much larger string values in Parquet files than the default StarRocks scanner buffer. When StarRocks reads the Parquet page and encounters a value exceeding 1,048,576 bytes, the C++ reader attempts to access memory outside its allocated block, triggering the

ERROR 5609

. References • administration/management/BE_configuration.md