Hey everyone I’m running into a strange issue with StarRocks 4.0 in the shared-data architecture. I’m seeing extremely high query latency due to IOTimeRemote. When I checked the GCS object storage metrics, I noticed the client error rate spikes up to about 50% (all of these errors were response “NOT_FOUND”)

it's a fresh deployed 4.0 cluster or upgraded from 3.x version?

It is a fresh deployed cluster

do you have the access log on gcs side, need to take a look what the access path pattern to these 404 not found

I checked the logs, all of the errors came from two access patterns: 1. It is checking for cluster snapshots, for some reason, it doesn’t find it, even though I checked the bucket and saw it 2. whenever I run a query, I get theses errors. Here is a sample: data-lakehouse/system_volume/6ef53bd0-4682-4f06-b8bc-d095455687a1/db13009/20742/20791/meta/0000000000005138_0000000000800055.meta

ok, we found similar pattern in v4.0.0, could be related to file_bundling feature introduced in v4.0

would 4.0.1 fix this issue?

4.0.1 already out, still looking into it.

Thank you Kevin! 🙏