Hi, I am using Theta Sketch in Druid to calculate Unique Count, I am using thetaSketch type to calculate unique count of one of our field named cnv_id. I am getting different results of unique counts for same time duration when I am changing the number of shards during ingestion. For example total number of unique count is 100 in the source fact when I am ingesting it with 1 shard then my theta sketch query column theta_sketch_estimate(ds_theta(cnv_id)) is coming 100 approx only but when I am using 4 shards to ingest then this value is coming 50. I am not sure what is the exact issue here and how to fix this. My hunch is sketches are maintained at shards level and seems to be not merging properly at segment layer. Can anyone help me out in this

The sketch is aggregated on a per-segment basis. However, if you're running multiple ingestion tasks each segment will generate its own segments. I am not as familiar with Theta sketches as I am with HLL, but generally speaking sketches have a hire error rate when estimating very low cardinality counts. The HLL sketch has some mechanisms to mitigate this, where it will use a different method to calculate low cardinality then high cardinality. I do not know if Theta sketches have a similar mechanism. My follow-up questions therefore are: 1. Is there a reason that you're using Theta vs HLL? Theta does support set operations that HLL does not, but for a simple count distinct HLL tends to be more efficient. 2. Is the 100 items a test case or realistic? If it is realistic, using an exact count distinct will probably perform better.

Answer to 1 question, when i was searching for unique count, I found theta sketch and it was fulfilling my requirements so didn't check HLL

Theta should be nearly exact at very low cardinalities

By "more efficient," I was referring to memory and storage consumption more than performance. I did not know about Theta sketch's ability to do early stopping, so that is great information. Thanks for lending your expertise @Jon 🙂