This message was deleted.
# troubleshootings
Slackbot
12/01/2022, 1:04 PMThis message was deleted.
g
Gian Merlino
12/01/2022, 5:03 PMthe best way is to do a test load and see how much space it takes up; compression can vary based on the dataset
Gian Merlino
12/01/2022, 5:05 PMa good rule of thumb is druid data is generally going to be slightly larger than parquet
Gian Merlino
12/01/2022, 5:05 PMso if your original is parquet you can use, like, 20–30% more than that as an estimate? (that will vary too; in both directions; so a test load is still useful)
Gian Merlino
12/01/2022, 5:06 PMnote that the compression is improving soon due to https://github.com/apache/druid/pull/12277
Gian Merlino
12/01/2022, 5:06 PMwill still be somewhat larger than parquet (we do have extra indexes, etc that parquet does not have) — but not as much larger as today
a
Abhishek Agarwal
12/02/2022, 8:21 AMAll great points by Gian. I will also point out that data locality matters too. Depending on how you are ingesting the data, you should use range partitioning (automatic if you are using SQL based ingest).
https://imply.io/blog/multi-dimensional-range-partitioning
While I have not directly verified this, range partitioning would make the dictionary compression (that Gian linked), even more effective. Because the dimensions with similar values are going to land in the same segment and the compression algorithm does much better when the values are similar.
Dictionary compression is going to be available in 25.0 which is just around the corner.
c
cactus
12/02/2022, 4:51 PMBrilliant! Thank you both very much for the valuable information!