@Rocky I am starting to use starrocks and we are doing some preliminary tests as in a future we want to migrate from hive to starrocks. I have a shared-data with 30 CN nodes and one FE. I have my cache at almost 0 as for benchmarking purposes I do not want the cache to pollute and I want to test hdfs against ceph to see which performs faster. I was creating a 5T table in ceph and in the compaction phase I found a checksum missmatch error. I tried to investigate it and I ended up activating a fallback configuration lake_clear_corrupted_cache_data = true whcih by default is off and this allowed finnishing the compaction of the table.

The checksum mismatch error you encountered during compaction in a shared-data (lake mode) cluster is a data integrity check failing on the local cache of the Compute Nodes (CN). Even if you set your cache size to "almost 0," StarRocks still uses the local storage directory (defined by

storage_root_path

) to manage metadata and temporary data during operations like compaction and loading. 1. Why did this happen? In shared-data mode, StarRocks performs compaction by reading data from remote storage (Ceph/HDFS), merging it on the CN, and writing new versions back to remote storage. * Cache Checksum: For performance, StarRocks often caches metadata and data blocks locally. When it reads these back from the local disk, it verifies them against a recorded CRC32 checksum. * Corruption: A "checksum mismatch" means the data read from the local disk did not match the expected checksum. This can be caused by: * Local disk hardware issues on one of your 30 CN nodes. * Filesystem-level errors (StarRocks performs best on XFS). * A known edge case (addressed in v3.4+) where cached data could become "confused" or inconsistent during high-concurrency operations. 2. The role of

lake_clear_corrupted_cache_data

By default, StarRocks stops the operation when a checksum mismatch occurs to prevent data corruption from propagating. * The Config: Setting

lake_clear_corrupted_cache_data = true

(introduced in v3.4) tells the CN: "If you find a corrupted data block in the local cache, don't fail the task. Instead, delete the corrupted cache file and re-fetch the data from the remote source (Ceph/HDFS)." * Result: This allowed your compaction to finish because the system effectively "self-healed" by bypassing the bad local cache and going back to the source of truth on Ceph. 3. Impact on Benchmarking Since your goal is to benchmark HDFS vs. Ceph: * Data Integrity: Enabling this flag is safe for your tests; it ensures that the "source of truth" (remote storage) is what ultimately gets used if the local cache fails. * Performance Noise: If this error happens frequently, the "self-healing" (deleting and re-downloading) will add latency to your Ceph benchmarks that wouldn't be present on a "healthy" run. You should check the CN logs (

<http://be.INFO|be.INFO>

or

be.WARNING

) to see if these errors are frequent or isolated to a specific node. * Vertical Compaction: Note that by default,

lake_enable_vertical_compaction_fill_data_cache

is

true

. Even with a small cache limit, compaction tasks may attempt to use the cache. Recommendations 1. Check CN Logs: Search for

checksum mismatch

in your CN logs to identify if a specific node is having local disk issues. 2. Filesystem: Ensure your CN

storage_root_path

is on XFS for best results. 3. Benchmarking Tip: To truly test raw remote storage performance without any cache interference, you can also try setting

datacache.enable = false

in your table properties, though the shared-data architecture is designed to perform best with the cache enabled. If you continue to see this error on Ceph but not HDFS, verify that the Ceph RGW (S3) interface is not returning partial or corrupted objects, though the error you described is almost always related to the local CN cache. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md