Hi Team, @Xiang Fu Need your suggestion in avoiding cpu when using DISTINCTCOUNTHLLPLUS( CASE_EXPRESSION). The Broker is encountering high cpu usage when deserializing the HyperLogLogPlus objects received from Servers. The flame graph shows majority(~~80%) of the CPU is consumed in reading the input stream. The call stack is HyperLogLogPlus.decodeBytes() -> Bits.getBits -> DataInputStream.readInt() -> ByteArrayInputStream.read().
I believe the Broker is receiving the serialized customObject type data from Server and needs to deserialize it into a HyperLogLogPLus object. Could you please suggest how can I reduce the CPU usage? Do we have any tunable configs to reduce the deserialization load on Broker (can server merge the HyperLogLog sketches!!) ? Thanks!
x
Xiang Fu
05/09/2025, 6:11 AM
do you have a groupby in the query? and the cardinality of the groups is huge?
Xiang Fu
05/09/2025, 6:12 AM
for hll object I think you can configure the log2m value to reduce the intermediate serializable object
@Xiang Fu Thank you for your quick response. Yes, my query has a groupby and the number of groups is usually high.
If I set higher value for log2m, is it expected to increase cpu usage in the servers? Just to get clarity, will the merging of the hll objects happen in the Server itself? Thanks!
x
Xiang Fu
05/09/2025, 6:16 AM
yes, both broker/server network and cpu for ser/de on both server and broker side. As well broker side merging intermediate objects
Xiang Fu
05/09/2025, 6:16 AM
merge will happen per group per server as well
g
Gyana Padhi
05/09/2025, 6:24 AM
As I understand the default value for log2m is 12. Do you have any recommended value for reducing the size of serializable object without significantly affecting the approximation? Thanks!