http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> getting this error ```E20260502 11:...
# questions-and-troubleshooting
n
@Rocky getting this error
Copy code
E20260502 11:59:52.598463 281471677759616 scan_operator.cpp:447] scan fragment 6d85945f-461e-11f1-8aeb-ca2daa10c36b driver 0 Scan tasks error: Internal error: BinaryDictPageDecoder::next_batch failed
be/src/storage/rowset/binary_dict_page.cpp:225 next_batch(read_range, dst)
be/src/storage/rowset/parsed_page.cpp:234 _data_decoder->next_batch(count, column)
be/src/storage/rowset/scalar_column_iterator.cpp:602 page_parse(values, &nread)
be/src/storage/rowset/segment_iterator.cpp:1730 _column_decoders[cid].decode_values_by_rowid(*ordinals, col.get())
be/src/storage/rowset/segment_iterator.cpp:1180 _finish_late_materialization(_context)
be/src/storage/lake/tablet_reader.cpp:257 _collect_iter->get_next(chunk)
be/src/connector/lake_connector.cpp:134 _prj_iter->get_next(chunk_ptr)
W20260502 11:59:52.598512 281471762301056 pipeline_driver.cpp:320] pull_chunk returns not ok status Internal error: BinaryDictPageDecoder::next_batch failed
be/src/storage/rowset/binary_dict_page.cpp:225 next_batch(read_range, dst)
be/src/storage/rowset/parsed_page.cpp:234 _data_decoder->next_batch(count, column)
be/src/storage/rowset/scalar_column_iterator.cpp:602 page_parse(values, &nread)
be/src/storage/rowset/segment_iterator.cpp:1730 _column_decoders[cid].decode_values_by_rowid(*ordinals, col.get())
be/src/storage/rowset/segment_iterator.cpp:1180 _finish_late_materialization(_context)
be/src/storage/lake/tablet_reader.cpp:257 _collect_iter->get_next(chunk)
be/src/connector/lake_connector.cpp:134 _prj_iter->get_next(chunk_ptr)
be/src/exec/pipeline/scan/scan_operator.cpp:250 _get_scan_status()
W20260502 11:59:52.598526 281471762301056 pipeline_driver_executor.cpp:175] [Driver] Process error, query_id=6d85945f-461e-11f1-8aeb-ca2daa10c367, instance_id=6d85945f-461e-11f1-8aeb-ca2daa10c36b, status=Internal e
r
Based on the error log you provided, it appears that StarRocks is encountering an internal failure while trying to decode a dictionary-encoded binary (string/varchar) column during a query scan. The stack trace points to 
BinaryDictPageDecoder::next_batch
 failing within the Lake (Shared-data) storage engine, specifically during the late materialization phase (
_finish_late_materialization
). Potential Causes 1. Late Materialization Bug: The error occurs during the late materialization step, where StarRocks reads the filtered row IDs for certain columns. There have been known issues in some versions where non-contiguous row reads in dictionary-encoded columns can cause decoding failures. 2. Data Cache Corruption (Lake Tables): Since the trace includes 
lake_connector.cpp
and 
tablet_reader.cpp
, this is a Shared-data (Lake) table. If the Data Cache on the BE/CN node is corrupted or inconsistent with the object storage (S3/HDFS), the decoder may receive malformed data, leading to this "Internal error." 3. Dictionary Decoding Issue: This can also happen if a column's dictionary page is missing or if the data indices in the page point to entries that don't exist in the dictionary (often due to a bug in compaction or schema change). 4. 4GB Binary Limit: In some recent versions (e.g., 3.3.x or 3.4.x), very large binary columns (exceeding 4GB in a single chunk) have been reported to cause issues in the dictionary decoder. Recommended Troubleshooting Steps You can try the following workarounds to bypass the issue and narrow down the cause: 1. Disable Late Materialization This is the most common workaround for errors occurring in 
_finish_late_materialization
. It forces the engine to read all columns at once rather than in two phases.
Copy code
sql
SET enable_late_materialization = false;
2. Disable Dictionary Optimization If the issue is specifically with the dictionary decoder, disabling this might force the engine to use a different decoding path.
Copy code
sql
SET enable_dictionary_optimize = false;
3. Bypass the Data Cache (For Lake Tables) If you suspect cache corruption, try running the query without the cache:
Copy code
sql
SET enable_scan_datacache = false;
If the query succeeds with this setting, you may need to clear the cache directory on your BE nodes. 4. Check for Data Corruption Try to identify if the error is specific to certain rows or a specific tablet: * Does the query fail every time? * Does it fail if you add a 
LIMIT
or filter for a specific time range? * Run 
SHOW TABLET <tablet_id>
(the tablet ID can usually be found in the BE logs preceding the error) to check the health of the data. Conclusion If disabling 
enable_late_materialization
or 
enable_scan_datacache
fixes the issue, it is likely a bug in the specific version you are running. I recommend checking if there is a newer patch release (e.g., if you are on 3.2.x, consider upgrading to the latest 3.2.x patch). If you can provide your StarRocks version (
SELECT VERSION();
), I can give more specific advice on known bugs for that release. References • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • unloading/Flink_connector.md • developers/build-starrocks/Build_in_docker.md
n
current version 3.3.9-dfae8f9
@Rocky still not working. can share command to clean cache
r
If you suspect that corrupted data in the Data Cache is causing the 
BinaryDictPageDecoder::next_batch
error, you can clear it using one of the following two methods. Method 1: Manual Cleanup (Recommended) This is the most thorough way to ensure the cache is completely purged. 1. Stop the BE/CN process. 2. Delete the cache directory. By default, the cache is stored in the 
datacache
folder within your storage path.
Copy code
bash
    # Replace /path/to/storage with your actual storage_root_path
    rm -rf /path/to/storage/datacache/*
Note: If you have multiple storage paths, repeat this for each one. 3. Start the BE/CN process. ────────── Method 2: Runtime Cleanup (No Restart Required) If you cannot restart the nodes, you can "force" a cleanup by setting the cache size to zero and then back to its original value. You will need to do this for each BE/CN node. 1. Find your BE/CN IDs:
Copy code
sql
    SHOW BACKENDS; -- or SHOW COMPUTE NODES;
2. Set the disk size to "0" (this triggers the system to purge existing cache files):
Copy code
sql
    -- Replace 10001 with your actual BE/CN ID
    UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size" AND BE_ID=10001;
3. Wait a few minutes, then set it back to your original size (e.g., 100GB, 2TB, etc.):
Copy code
sql
    UPDATE be_configs SET VALUE="200G" WHERE NAME="datacache_disk_size" AND BE_ID=10001;
────────── Additional Step: Refresh Metadata If the issue is related to a Shared-data (Lake) table and cleaning the Data Cache doesn't help, the metadata itself might be stale. Try forcing a metadata refresh for the table:
Copy code
sql
REFRESH EXTERNAL TABLE <table_name>;
-- For internal Lake tables, you can also try:
RECOVER TABLE <table_name>;
If the error persists... If clearing the cache and disabling late materialization (
SET enable_late_materialization = false;
) both fail, this is likely a software bug related to how that specific version handles dictionary-encoded strings. * Check your version: 
SELECT VERSION();
* If you are on an older version of 3.1.x or 3.2.x, this is a known issue that was fixed in later patch releases (e.g., 3.2.4+). I recommend upgrading to the latest maintenance release of your current branch. References • data_source/data_cache_troubleshooting.md • administration/management/BE_configuration.md • faq/Others.md
n
@Rocky this did not fix
r
Since the previous steps (disabling late materialization and clearing the data cache) did not resolve the issue, we are likely dealing with a more specific dictionary-decoding bug or a configuration conflict within the Lake (Shared-data) storage engine. The error 
BinaryDictPageDecoder::next_batch failed
indicates that the query engine is failing to map row indices to their corresponding string values in the dictionary page. Please try the following session variables in order. These target the specific decoding optimizations that remain active even when late materialization is off. 1. Disable Low Cardinality Optimization StarRocks has a specialized optimization for string columns with few unique values. If there is a bug in the global dictionary builder for this query, it can cause this decoder failure.
Copy code
sql
SET enable_low_cardinality_optimize = false;
2. Disable Dictionary-based Filter Pushdown This prevents the engine from trying to optimize filters directly against the dictionary pages, which is where the 
BinaryDictPageDecoder
is most active.
Copy code
sql
SET enable_dictionary_optimize = false;
3. Check for specific Column Corruption To isolate the issue, try to identify if a specific column is causing the crash: _ Try running 
SELECT count(_) FROM your_table;
. If this succeeds, the metadata is likely fine, and the issue is with a specific data column. * Try selecting columns one by one to find the "broken" one. Usually, it is a 
VARCHAR
or 
STRING
column. 4. Check StarRocks Version & Known Bugs This specific error (
binary_dict_page.cpp:225
) is associated with a few known issues in older 3.x releases: * Version 3.1.x / 3.2.x: There was a bug where sparse row ID ranges (caused by highly selective filters) caused the dictionary decoder to calculate offsets incorrectly. This was fixed in 3.1.10+ and 3.2.4+. * Version 3.3.x: A similar issue appeared with very large strings (close to or exceeding 4GB in a single scan block). Could you run 
SELECT VERSION();
and share the output? If you are on an early 3.x release, the only permanent fix may be a patch upgrade to the latest maintenance release of your branch. 5. Last Resort: Force Compaction If the data was written in a "corrupt" state (e.g., during a node failure or a specific version bug), you can try to force a compaction on the affected tablets to rewrite the dictionary pages:
Copy code
sql
-- You would need the tablet ID from the BE logs
SUBMIT TASK AS ALTER TABLE <table_name> COMPACT;
(Note: Compaction for Lake tables is managed by the FE/Leader node. Check 
SHOW PROC '/compactions';
to see the status.) References • administration/management/BE_configuration.md • faq/Sql_faq.md • faq/shared_data_faq.md • release_notes/release-3.2.md References • data_source/datalake_faq.md • administration/management/BE_configuration.md • faq/shared_data_faq.md • faq/loading/Loading_faq.md • administration/Meta_recovery.md
Open in Slack
PreviousNext
Powered by