Hi, I am creating a custom druid metric column whi...
# devb
birinder tiwana
03/14/2024, 4:46 PMHi, I am creating a custom druid metric column which will store a custom object and serialize it to a byte []. Size estimation is ~1KB. If within my ingestion spec i specify the metricCompression = lz4 (the default one), will the data in this particular custom metric column also get compressed? The documentation on this is a bit unclear because it says "Compression format for primitive type metric columns" Looking at some of the code it doesn't seem like there would be any compression
https://github.com/apache/druid/blob/master/processing/src/main/java/org/apache/druid/segment/IndexMergerV9.java#L711-L716
c
Clint Wylie
03/15/2024, 3:58 AMyou need to provide the serializer/deserializer yourself to get compression
Clint Wylie
03/15/2024, 3:59 AMthere are a handful of things you could use to achieve this without writing the whole thing
Clint Wylie
03/15/2024, 4:03 AMjson columns use a thing called
CompressedVariableSizedBlobColumnSerializerCompressedVariableSizedBlobColumnSupplierComplexMetricSerde🙌 1
Clint Wylie
03/15/2024, 4:03 AMif your values are fixed width, you could also use the stuff that
CompressedVariableSizedBlobColumnSerializerCompressedVariableSizedBlobColumnSupplierClint Wylie
03/15/2024, 4:05 AMClint Wylie
03/15/2024, 4:06 AMthere is also some different stuff used by things like the compressed bigdecimal extension, which i personally find a bit more complicated to follow, but is also used by some things, can find it in here https://github.com/apache/druid/tree/master/processing/src/main/java/org/apache/druid/segment/serde/cell
Clint Wylie
03/15/2024, 4:09 AMim a bit less familiar with it but can follow from here https://github.com/apache/druid/blob/master/extensions-contrib/compressed-bigdecim[…]druid/compressedbigdecimal/CompressedBigDecimalMetricSerde.java
s
Samarth Jain
03/15/2024, 11:03 PMWe did implement our own serde when we implemented SpecatorHistogram extension in Druid.
See - https://github.com/apache/druid/blob/master/extensions-contrib/spectator-histogram[…]d/spectator/histogram/SpectatorHistogramComplexMetricSerde.java
🙌 1
b
birinder tiwana
03/18/2024, 7:11 PMThanks a lot for all the inputs!! Really appreciate the guidance here. Since we will need to store the timestamp column along with this byte[] and then do custom aggregations/post aggregations etc. based on that timestamp, we have decided to implement our own extension. We will serialize/deserialize the actual byte[] within our app layer itself (probably can be done at druid also but that's besides the point here) because we don't need to open/index it within druid store.