I got a question regarding observed disk read spee...
# generalm
maxidl
11/03/2025, 7:17 PMI got a question regarding observed disk read speed. For the operation shown in the screenshot, the sword fish progress shows me 707MiB bytes read, which is very much on point. However, in htop i always see >1.5G/s DISK READ while this operation is going on, which does not really make sense to me. The input data (all 150M rows) is about 4.5GB compressed parquet (the column I am reading). So seeing read speeds of >1.5G/s for a minute does not really make sense. Does anyone know what leads to such high read speed being shown in htop?
c
Colin Ho
11/04/2025, 9:12 AMcan you try doing
df.explain(True)Colin Ho
11/04/2025, 9:14 AMand also can you check if this is the same behaviour with just a
read_parquet(...).collect()37 Views