Hi team, We are running StarRocks 3.3.18 (shared-n...
# questions-and-troubleshooting
m
Hi team, We are running StarRocks 3.3.18 (shared-nothing mode) on a Kubernetes cluster using SSD storage. The cluster has 10 backends and 3 frontends, deployed using StarRocks Operator v1.11.1. We’re ingesting data at around 1 million rows/sec from a Kafka topic into a database with duplicate key tables (replication factor 3) using routine load. Occasionally, when querying the data (either during or after ingestion), we get the following error: Bad page: too small size (0), file(/opt/starrocks/be/storage/data/224/3887247/1220709102/020000000357739ee94bd493e75f6dc19130aee3585dd9ab_0.dat) On retry, the query succeeds by fetching data from another replica. After checking the source code, we found this error originates from
page_io.cpp
, which indicates the page size is less than 8 bytes (4-byte footer + 4-byte checksum). However, when we run
CHECK CONSISTENCY
or
REPAIR
commands, no issue is detected, and the cluster reports the tablet as healthy. Even deleting the corrupted
.dat
file manually does not trigger any action from consistency check or repair. The only effective fix so far is marking the affected tablet as bad manually on that BE node. Questions: What exactly does “page” refer to in this context — is it a logical data page or a physical block in the
.dat
file? Why does
CHECK CONSISTENCY
not detect these small-size pages? Is there any recommended way to proactively detect such “bad pages” before they cause query errors? Any insights or debugging directions would be greatly appreciated 🙏