< Rocky> I have a hudi table with varchar <tel 1073741824|10 StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> I have a hudi table with varchar(<t...

Mohan

07/31/2026, 7:31 AM

@Rocky I have a hudi table with varchar(1073741824) with hive as metastore connected as unified external catalog to starrocks shared data architecture. It is giving error

Copy code

ERROR 5609 (22000): going to read out-of-bounds data, offset=1048799,count=560,size=1048799: file = gs:

as starrocks has varchar limit of 1MB and data could be more than that. How can I read data properly without losing data ?

Rocky

07/31/2026, 7:32 AM

The error you are encountering occurs because StarRocks has a hard limit of 1 MB (1,048,576 bytes) for the

VARCHAR

data type in its internal processing engine (BE). Even though your Hudi/Hive table defines the column as a much larger

VARCHAR

(1GB), the StarRocks reader attempts to process it within this 1MB ceiling. When a single value in a Parquet/Hudi file exceeds this size, the reader hits a memory boundary, resulting in the "out-of-bounds" error. To read this data without losing it, you can use the following strategies: 1. Map or Cast to the
JSON
Type (Up to 16 MB) In StarRocks, the

JSON

data type has a higher limit of 16 MB per value. If your large string data is under 16 MB, you can bypass the

VARCHAR

limit by treating the column as JSON. * Create a View with a Cast: You can create a view on top of your Hudi table that casts the problematic column to the

JSON

type. This often allows the scanner to handle the larger payload.

Copy code

sql
    CREATE VIEW hudi_large_data_view AS
    SELECT
        id,
        CAST(large_string_col AS JSON) AS large_string_col
    FROM hudi_catalog.db.table;

Note: If the Parquet scanner fails before the cast can be applied, this might still trigger the error. If so, see the next step. 2. Load into an Internal Table with
JSON
Type If you need to query this data frequently and the View approach fails, you can load the data into a StarRocks internal table where the column is explicitly defined as

JSON

Copy code

sql
-- Create internal table
CREATE TABLE internal_table (
    id INT,
    large_col JSON
) ENGINE=OLAP
DISTRIBUTED BY HASH(id);

-- Load data
INSERT INTO internal_table 
SELECT id, CAST(large_string_col AS JSON) 
FROM hudi_catalog.db.table;

3. Check and Truncate (If data > 16 MB) If your data truly exceeds 16 MB for a single row, StarRocks currently does not have a native string type to support it (as of version 3.x). You may need to: * Truncate during read: Use

SUBSTRING

if you only need the beginning of the text.

Copy code

sql
    SELECT SUBSTRING(large_string_col, 1, 1048576) FROM hudi_catalog.db.table;

* Pre-process the data: In your upstream Hudi/Spark job, split the large text into multiple rows or smaller chunks before StarRocks reads it. 4. BE Configuration (Limited Effect) There is a BE configuration

enable_check_string_lengths

(default:

true

), but this is primarily for preventing compaction failures from existing out-of-bound data in internal tables rather than allowing larger reads from external catalogs. Summary of Limits Type | Max Size | Best Use Case VARCHAR | 1 MB | Standard text data STRING | 1 MB | Alias for VARCHAR(65533) or 1MB in newer versions JSON | 16 MB | Semi-structured or very large text (workaround for > 1MB) Recommendation: Try the

CAST(... AS JSON)

approach first, as it is the most common workaround for strings between 1MB and 16MB. If the data is consistently larger than 16MB, StarRocks is not currently optimized to hold those values in a single column. References • data_source/catalog/maxcompute_catalog.md • using_starrocks/Flat_json.md • sql-reference/sql-functions/json-functions/json-query-and-processing-functions/to_json.md • data_source/catalog/hive_catalog.md • sql-reference/data-types/semi_structured/JSON.md • data_source/External_table.md

Mohan

07/31/2026, 7:34 AM

@Rocky I'm trying to insert into starrocks table as json field only but when reading varchar field giving the out of bound error

Rocky

07/31/2026, 7:34 AM

The error occurs because StarRocks has a hard limit of 1 MB (1,048,576 bytes) for the

VARCHAR

and

STRING

data types within its execution engine (BE). When you use a Unified External Catalog (like Hudi or Hive), StarRocks automatically maps Hive's

STRING

VARCHAR

to its own

VARCHAR

type. If a single value in your Hudi Parquet files exceeds this 1 MB limit, the Parquet scanner fails with the "out-of-bounds" error during the read phase. To read data up to 16 MB without loss, you must treat the column as the JSON
data type, which has a higher internal limit. Since the External Catalog auto-maps the type to

VARCHAR

, you need to bypass this auto-mapping using one of the following methods: Method 1: Use the
FILES()
Table Function (Recommended for 3.1+) The

FILES()

function allows you to manually define the schema of the files you are reading. By defining the problematic column as

JSON

instead of

VARCHAR

, you enable the scanner to handle values up to 16 MB.

Copy code

sql
INSERT INTO starrocks_internal_table
SELECT * FROM FILES(
    "path" = "<gs://your_bucket/path/to/hudi_table/*/*.parquet>", -- Path to Hudi parquet files
    "format" = "parquet",
    "schema" = "id INT, large_string_col JSON, other_col VARCHAR(255)", -- Manually define schema
    "gs.access_key" = "...",
    "gs.secret_key" = "..."
);

Note: This method reads raw Parquet files and does not process Hudi-specific metadata (like Merge-On-Read logs). It works best for Copy-On-Write tables. Method 2: Manual External Table (Legacy approach) Instead of using the automated Catalog, you can create a manual Hudi External Table. In the manual definition, you can attempt to "coerce" the column type by defining it as

JSON

. 1. Create a Resource:

Copy code

sql
   CREATE EXTERNAL RESOURCE "hudi_resource"
   PROPERTIES (
       "type" = "hudi",
       "hive.metastore.uris" = "<thrift://your-hms:9083>"
   );

2. Create the Table with
JSON
type:

Copy code

sql
   CREATE EXTERNAL TABLE manual_hudi_table (
       id BIGINT,
       large_string_col JSON  -- Force JSON type here
   ) ENGINE=HUDI
   PROPERTIES (
       "resource" = "hudi_resource",
       "database" = "your_hudi_db",
       "table" = "your_hudi_table"
   );

Now,

SELECT * FROM manual_hudi_table

will use the 16 MB limit for that column. Method 3: Truncate Data (If data > 16 MB) If your data truly exceeds 16 MB, StarRocks currently cannot store it in a single column. You would need to truncate the data during the read using

SUBSTRING

, but this only works if the scanner doesn't fail first. If the scanner fails, you must use one of the methods above first to "get the data into the engine" as a

JSON

type, and then truncate it:

Copy code

sql
SELECT SUBSTRING(CAST(large_string_col AS STRING), 1, 1048576) 
FROM manual_hudi_table;

Summary of Workaround Feature | Limit | Why your current approach fails VARCHAR | 1 MB | Catalog auto-maps Hive STRING to SR VARCHAR(1MB). JSON | 16 MB | The scanner supports 16MB for this type specifically. Important Note: If you are using Merge-On-Read (MOR) tables and need to read the latest view (including log files), Method 2 is required as

FILES()

only sees the base Parquet files. References • sql-reference/sql-functions/table-functions/files.md

Mohan

07/31/2026, 7:35 AM

I can't truncate any data @Rocky

Rocky

07/31/2026, 7:35 AM

The error you are seeing occurs because StarRocks has a hard limit of 1 MB (1,048,576 bytes) for the

VARCHAR

and

STRING

data types in its execution engine. In a Unified External Catalog (Hudi/Hive), StarRocks automatically maps the source

STRING

to its internal

VARCHAR(1048576)

. When the Parquet scanner encounters a single field value larger than this, it hits a memory boundary and throws the "out-of-bounds" error. To read this data without losing it, you can use the following methods to increase the limit to 16 MB. Method 1: Use a Manual External Table (Up to 16 MB) The automated Hudi Catalog (Unified Catalog) does not allow you to change the type mapping. However, you can create a Manual Hudi External Table where you explicitly define the problematic column as the

JSON

data type. StarRocks'

JSON

type supports up to 16 MB per value. 1. Create the Hudi Resource:

Copy code

sql
    CREATE EXTERNAL RESOURCE "hudi_res"
    PROPERTIES (
        "type" = "hudi",
        "hive.metastore.uris" = "thrift://<your_hms_host>:9083"
    );

2. Create the Manual External Table: Explicitly map the large VARCHAR column to `JSON`:

Copy code

sql
    CREATE EXTERNAL TABLE manual_hudi_table (
        id BIGINT,
        large_col JSON,  -- Map the 1GB field to JSON here
        other_fields...
    ) ENGINE=HUDI
    PROPERTIES (
        "resource" = "hudi_res",
        "database" = "your_db",
        "table" = "your_hudi_table"
    );

Now, when you query

manual_hudi_table

, the scanner will allow up to 16 MB for that column. Method 2: Use the
FILES()
Table Function (Up to 16 MB) If you are using StarRocks 3.1 or later, you can use the

FILES()

function to query the Parquet files directly. This allows you to define the schema on the fly and treat the large column as

JSON

Copy code

sql
SELECT * FROM FILES(
    "path" = "<gs://your_bucket/path/to/hudi_table/*/*.parquet>",
    "format" = "parquet",
    "schema" = "id INT, large_col JSON, date_col DATE",
    "gs.access_key" = "...",
    "gs.secret_key" = "..."
);

Important Limitation: Data > 16 MB If any single row in your data exceeds 16 MB, StarRocks currently cannot read it into a single column. The engine's internal block format is not designed to handle single-field values of that size (like 1GB). If your data is truly larger than 16 MB and you cannot truncate it: 1. Pre-processing (Recommended): You must use an upstream tool (like Spark or Flink) to split the large string into multiple rows or an

ARRAY<VARCHAR>

before StarRocks reads it. 2. Truncate to 16MB: If you can accept only the first 16MB, you can cast the column to

JSON

using the methods above and then use

SUBSTRING(CAST(col AS STRING), 1, 16777216)

. Summary * Default: 1 MB limit (Error 5609). * Manual Mapping to `JSON`: 16 MB limit (Best workaround). * Above 16 MB: Not supported in a single field; requires upstream splitting. References • quick_start/hudi.md • data_source/catalog/hudi_catalog.md • data_source/feature-support-data-lake-analytics.md

2 Views

Open in Slack

Previous Next