Hi All, Is there a way we can measure the entire column size stored in the data source ? For example we want to find out which column is occupying more space in S3/Historicals ?

First get the segment from deep storage to a local disk for which you want to check the column size. Unzip the segment inside the zip file there should be a file called Meta.smoosh which will have the metadata. The 3rd and 4th parameter is the start and end offset So we can calculate the size (which is in bytes) by doing “end location - start location” for a given column. In the smoosh file the sequence of columns might differ than how you see the columns in the query, so check the query and then correlate with the smoosh file.

The

dump-segment

tool can help with this https://druid.apache.org/docs/latest/operations/dump-segment/ - if you use the

metadata

option https://druid.apache.org/docs/latest/operations/dump-segment/#metadata-dumps it includes some info about the column sizes

Thanks @Kyle Hoondert @Bibek Sahoo will check these out