We have a CN cluster (shared-data) with 40 kubernetes pods 15 of the 40 pods have this error continuously:

W20251029 06:20:32.674981 123762826905152 cache_file.h:130] write star cache file data/0000000000442c19_d6948c74-e5d6-4666-b02d-57395b6712dd.dat failed, errmsg: INTERNAL: fail to write block because no valid cache space

For those 15 pods, we have this error in the UI

Status: Normal, DiskUsage: 0B/0B, MemUsage: 0B/0B

For the 25 pods without the above error, they have this:

Status: Normal, DiskUsage: 30.8GB/140GB, MemUsage: 0B/0B

We had not set any datacache setting yet But was wondering why some pods have the no valid cache space error, and others don't? Running CN 3.5.7

have you provided pvc for storage root path as the cache disk?

No we did not (yet), we are the in process of setting it up To resolve we set

datacache_disk_size = 120G

and the errors went away