http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> `execute task mv-6373980 failed: R...
# questions-and-troubleshooting
a
@Rocky 
execute task mv-6373980 failed: Refresh mv mv_ship failed after 1 times, try lock failed: 0, error-msg : com.starrocks.common.DdlException: fail to find valid type encoding, type:BYTE_ARRAY, encoding:DELTA_LENGTH_BYTE_ARRAY: file = <s3a://datalake/tst/test2/data/20260806_154638.parquet>: BE:10003
  
at com.starrocks.common.ErrorReport.reportDdlException(ErrorReport.java:100)
  
at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:2770)
[wrapped] com.starrocks.common.StarRocksException: fail to find valid type encoding, type:BYTE_ARRAY, encoding:DELTA_LENGTH_BYTE_ARRAY: file = <s3a://datalake/tst/test2/data/20260806_154638.parquet>: BE:10003
  
at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:3006)
  
at com.starrocks.load.InsertOverwriteJobRunner.executeInsert(InsertOverwriteJobRunner.java:409)
  
at com.starrocks.load.InsertOverwriteJobRunner.doLoad(InsertOverwriteJobRunner.java:178)
  
at com.starrocks.load.InsertOverwriteJobRunner.handle(InsertOverwriteJobRunner.java:158)
  
at com.starrocks.load.InsertOverwriteJobRunner.transferTo(InsertOverwriteJobRunner.java:224)
  
at com.starrocks.load.InsertOverwriteJobRunner.prepare(InsertOverwriteJobRunner.java:277)
  
at com.starrocks.load.InsertOverwriteJobRunner.handle(InsertOverwriteJobRunner.java:155)
  
at com.starrocks.load.InsertOverwriteJobRunner.run(InsertOverwriteJobRunner.java:143)
  
at com.starrocks.load.InsertOverwriteJobMgr.executeJob(InsertOverwriteJobMgr.java:90)
  
at com.starrocks.qe.StmtExecutor.handleInsertOverwrite(StmtExecutor.java:2522)
  
at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:2615)
  
at com.starrocks.qe.StmtExecutor.handleDMLStmtWithProfile(StmtExecutor.java:2531)
  
at com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.refreshMaterializedView(PartitionBasedMvRefreshProcessor.java:1255)
  
at com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedView(PartitionBasedMvRefreshProcessor.java:502)
  
at com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedViewWithRetry(PartitionBasedMvRefreshProcessor.java:408)
  
at com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doMvRefresh(PartitionBasedMvRefreshProcessor.java:342)
  
at com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.processTaskRun(PartitionBasedMvRefreshProcessor.java:195)
  
at com.starrocks.scheduler.TaskRun.executeTaskRun(TaskRun.java:382)
  
at com.starrocks.scheduler.TaskRunExecutor.lambda$executeTaskRun$0(TaskRunExecutor.java:60)
  
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  
at java.base/java.lang.Thread.run(Thread.java:840)
r
The error 
fail to find valid type encoding, type:BYTE_ARRAY, encoding:DELTA_LENGTH_BYTE_ARRAY
indicates that StarRocks' native C++ Parquet reader encountered a file using an encoding it does not yet support. Cause The 
DELTA_LENGTH_BYTE_ARRAY
 encoding is a standard Parquet feature (often used by Databricks, Spark, or newer Parquet writers) that was historically not implemented in the StarRocks native backend (BE) reader. When the native reader encounters this encoding in a 
BYTE_ARRAY
(String/Binary) column, it fails with the error you provided. Solutions 1. Upgrade StarRocks (Recommended) Support for 
DELTA_LENGTH_BYTE_ARRAY
was added in recent versions of StarRocks (see PR #57954). * Version: This support is available in StarRocks 3.4.0 and later, as well as recent patch releases of the 3.3.x series. If you are on an older version, upgrading will resolve this issue for the native reader. 2. Workaround: Use the JNI Reader If you cannot upgrade immediately, you can force StarRocks to use the JNI-based Parquet reader. This reader uses the Java 
parquet-mr
library, which supports a wider range of encodings than the native C++ reader. To apply this to your Materialized View (MV) refresh, you can set the 
force_jni_scanner
session variable as a property of the MV. This ensures that background refresh tasks will use the JNI reader:
Copy code
sql
ALTER MATERIALIZED VIEW mv_ship SET ("session.force_jni_scanner" = "true");
After setting this property, trigger the refresh again:
Copy code
sql
REFRESH MATERIALIZED VIEW mv_ship;
3. One-time Manual Refresh If you want to test the fix or perform a one-time manual refresh from your current session:
Copy code
sql
SET force_jni_scanner = true;
REFRESH MATERIALIZED VIEW mv_ship WITH SYNC MODE;
Summary of what to check: * StarRocks Version: Check your version with 
SELECT version();
. If it is older than 3.3.x (late patches) or 3.4.0, the native reader lacks this support. * External Data: Since the file is on S3 (
s3a://...
), verify if the tool producing these Parquet files (e.g., Spark, Flink) can be configured to use 
PLAIN
encoding for compatibility, though using the JNI reader or upgrading StarRocks is generally the better. References • administration/management/BE_configuration.md • faq/Deploy_faq.md • deployment/environment_configurations.md • sql-reference/sql-functions/JAVA_UDF.md • administration/management/FE_configuration.md
a
@Rocky Starrocks version is 3.5.16. The parquet files are starrocks own files - nothing new is created from beyond starrocks. It has a problem reading its own file. When I tried to apply the setting you gave, I am getting: 
Getting analyzing error. Detail message: Unknown system variable 'force_jni_scanner', the most similar variables are {'skew_join_rand_range', 'pq_refine_factor', 'orc_use_column_names'}.
r
The error 
fail to find valid type encoding, type:BYTE_ARRAY, encoding:DELTA_LENGTH_BYTE_ARRAY
confirms that the native C++ Parquet reader in the StarRocks Backend (BE) encountered a file using an encoding it could not decode. Although support for 
DELTA_LENGTH_BYTE_ARRAY
was added in StarRocks 3.4.0 (via PR #57954), you are still seeing this in version 3.5.16. This can happen if the encoding is applied to a specific data type or nested structure that the native reader's implementation doesn't yet cover, or if there is a regression. 1. Correct Variable to Enable JNI Reader The variable 
force_jni_scanner
is not a standard system variable in StarRocks 3.5. To force StarRocks to use the Java-based reader (which has broader encoding support), use the following session variable:
Copy code
sql
-- Enable the JNI-based scanner for external data formats
SET enable_jni_scanner = true;
To apply this specifically to your Materialized View (MV) so that background refreshes use it:
Copy code
sql
ALTER MATERIALIZED VIEW mv_ship SET ("session.enable_jni_scanner" = "true");
2. Why StarRocks Created Incompatible Files You mentioned StarRocks created these files. If you used 
INSERT INTO FILES()
or 
EXPORT
to generate them, StarRocks uses its Arrow-based writer. In newer versions, this writer may default to Parquet 2.0 encodings (like 
DELTA_LENGTH_BYTE_ARRAY
) to optimize string storage. If you want to ensure StarRocks generates files that its own native reader can always handle without JNI, you should specify the Parquet version and encoding style during the write/unload process:
Copy code
sql
-- Example when using INSERT INTO FILES() to write the data
INSERT INTO FILES (
    "path" = "s3a://...",
    "format" = "parquet",
    "parquet.version" = "1.0",           -- Forces V1 compatibility
    "parquet.use_legacy_encoding" = "true" -- Uses older, more compatible encodings
) SELECT ...;
3. Summary of Steps 1. Quick Fix: Run 
SET enable_jni_scanner = true;
before refreshing the MV to see if the JNI reader bypasses the error. 2. Permanent MV Fix: Use the 
ALTER MATERIALIZED VIEW ... SET ("session.enable_jni_scanner" = "true")
command to persist the setting for that MV. 3. Root Cause Investigation: If the data contains complex types (like nested 
MAP
or 
STRUCT
), the native reader has more limitations than the JNI reader. Check the schema of the source files. If 
enable_jni_scanner
is also not recognized, please run 
SHOW VARIABLES LIKE '%jni%';
to find the specific variable name enabled in your patch version (e.g., 
avro_use_jni_reader
is common, but 
enable_jni_scanner
is the standard toggle for most external catalog/file in-catalogs since v3.1). 3.10). • administration/management/BE_configuration.md • sql-reference/System_variable.md • sql-reference/sql-functions/table-functions/files.md • release_notes/release-3.4.md • [release_notes/release-3.5.md](https://docs.starrocks.io/docs/release_notes/release References • sql-reference/System_variable.md
a
@Rocky - response incorrect. There is no enable_jni_scanner variable either at all. The only jni variables starrocks 3.5.16 recognizes are hudi_mor_force_jni_reader and paimon_force_jni_reader Need a different suggestion.
r
Since you are on version 3.5.16, the 
DELTA_LENGTH_BYTE_ARRAY
encoding should be supported by the native reader, but the error indicates a failure in the decoding logic (possibly due to a bug in the 3.5.x implementation or an edge case with the specific data type). Because 
enable_jni_scanner
and 
force_jni_scanner
are not recognized in your environment, StarRocks likely uses catalog-specific or feature-specific toggles to control the reader type. 1. Identify and Disable the Native Reader for your Catalog The fallback to the JNI (Java) reader is handled differently depending on which catalog type your Materialized View is reading from. Identify the base table's catalog and use the corresponding session variable: * For Hive Catalog:
Copy code
sql
    SET hive_parquet_use_native_reader = false;
* For Iceberg Catalog:
Copy code
sql
    SET iceberg_parquet_use_native_reader = false;
* For Delta Lake Catalog:
Copy code
sql
    SET deltalake_parquet_use_native_reader = false;
To make this permanent for your Materialized View:
Copy code
sql
ALTER MATERIALIZED VIEW mv_ship SET ("session.hive_parquet_use_native_reader" = "false");
-- (Replace 'hive' with 'iceberg' or 'deltalake' as appropriate)
2. Disable 3.5-specific Parquet Optimizations StarRocks 3.5.0 introduced new Parquet reading optimizations (Bloom Filter and Page Index). It is possible the error is triggered by one of these new code paths. Try disabling them before refreshing:
Copy code
sql
SET enable_parquet_reader_page_index = false;
SET enable_parquet_reader_bloom_filter = false;
3. Check for Global "Native" Toggle If none of the above exist, check for a general native-reader switch by running:
Copy code
sql
SHOW VARIABLES LIKE '%native%';
Look for 
enable_native_reader
or 
use_native_parquet_reader
. If found, set it to 
false
. 4. Root Cause: How StarRocks created these files If StarRocks created these files using 
INSERT INTO FILES()
or 
EXPORT
, it likely used the newer Parquet 2.0 format which defaults to 
DELTA_LENGTH_BYTE_ARRAY
for strings. To prevent this in the future, you can force StarRocks to write "Legacy" or "V1" Parquet files that are guaranteed to be compatible with all readers: When writing the data that the MV eventually reads, use these properties:
Copy code
sql
INSERT INTO FILES (
    "path" = "s3a://...",
    "format" = "parquet",
    "parquet.use_legacy_encoding" = "true",
    "parquet.version" = "1.0"
) SELECT ...;
Recommendation 1. Run 
SHOW VARIABLES LIKE '%native%';
to find the correct toggle for your version. 2. If it is a Hive or Iceberg table, disabling the native reader (Step 1) is the most reliable way to bypass the C++ reader error. 3. If this is a regression in 3.5.16, I recommend reporting this to the StarRocks community with the file schema, as the native reader is intended to support this encoding in your version. References • sql-reference/System_variable.md • administration/management/BE_configuration.md • sql-reference/sql-functions/table-functions/files.md • release_notes/release-3.5.md • data_source/catalog/deltalake_catalog.md • data_source/catalog/iceberg_catalog.md • data_source/catalog/hive_catalog.md • release_notes/release-3.4.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md
2 Views
Open in Slack
PreviousNext
Powered by