I noticed that my hll sketches are not updating (v...
# general
m
I noticed that my hll sketches are not updating (values are not increasing). In addition to adding logs, I'm looking at the sketches themselves. When I translated one to a string, I got:
Copy code
### HLL SKETCH SUMMARY: 
  Log Config K   : 12
  Hll Target     : HLL_4
  Current Mode   : SET
  Memory         : false
  LB             : 11.0
  Estimate       : 11.000000273187965
  UB             : 11.000549494958403
  OutOfOrder Flag: false
  Coupon Count   : 11
Does anyone know what the mode/memory/out of order flag means?
this one has current mode of LIST:
Copy code
### HLL SKETCH SUMMARY: 
  Log Config K   : 12
  Hll Target     : HLL_4
  Current Mode   : LIST
  Memory         : false
  LB             : 1.0
  Estimate       : 1.0
  UB             : 1.000049929250618
  OutOfOrder Flag: false
  Coupon Count   : 1
and the only example I've found has current mode
HLL
https://datasketches.apache.org/docs/HLL/HllJavaExample.html
j
the sketch moves through several modes to minimize memory use
it starts with a list of hashes, then moves to a hash set once there are … not many. finally it moves to HLL mode when the memory of storing the set would exceed the HLL array size
memory is whether it’s an on-heap object or lives in “off-heap” memory outside the reach of the JVM’s garbage collector
false = regular object
and out of order impacts the estimation algorithm. there’s a better estimator you can use, but it requires having seen all the data in a single stream. that means it doesn’t work if you’ve merged sketches in HLL mode since you’ve mixed the order of arrival of events in the merge process.
but what exactly is the issue? a sketch in HLL mode will not necessarily change on every update
m
thanks for the explanation! the issue I'm having is that I'm sending druid data that should increase cardinality counts, and the data is definitely stored in druid, but the counts aren't increasing
j
i just know the actual sketch details, unfortunately
m
thanks anyways!