This message was deleted.
# generals
Slackbot
09/28/2023, 10:50 AMThis message was deleted.
j
Jon
09/28/2023, 2:40 PMIt is an approximate count. For 2^16, 2 standard deviations is roughly +-0.78%. Are you consistently seeing larger deviations than that?
a
Anirudh Deshpande
09/28/2023, 3:01 PMYes. I am seeing close to 5% while doing a UNION. With more set operations, the error is increasing.
l
Lee Rhodes
09/29/2023, 4:44 PMWhat kind of set operations? Are you doing any intersection or set difference operations?
Lee Rhodes
10/02/2023, 4:26 PMWe would like to help you troubleshoot this error issue, but without more information from you there is not much we can do.
w
William Kennedy
10/02/2023, 5:28 PMwe encountered this problem it is computing a sketch of the sketch with druid sql
William Kennedy
10/02/2023, 5:28 PMit should be a known bug, but idk if it’s matriculated up
William Kennedy
10/02/2023, 5:29 PMyou can get around it with a special function on ingest. Tag the column as a string in the ingest spec and then use COMPLEX_DECODE_BASE64(‘thetaSketch’,active_users_1d_new_theta)
a
Alexander Saydakov
10/02/2023, 5:44 PMshouldn’t the results look completely arbitrary in such a case, not just a few percent off?
j
Jon
10/02/2023, 5:53 PMIf the input is all mostly single-item sketches I could see something weird like that being possible
a
Alexander Saydakov
10/02/2023, 6:11 PMyes, and most likely in that case we should see an underestimate since sketches representing several distinct values will be counted as one distinct value
➕ 1
w
William Kennedy
10/02/2023, 6:23 PMyeah it should always be an order of magnitude underestimate and was when I tested it
l
Lee Rhodes
10/02/2023, 6:30 PMSo if you know it is going to be a gross underestimate, I don’t understand your issue that you observe 5% error. Sketching sketches as input is NOT a Union operation.
j
Jon
10/02/2023, 6:32 PMI believe @William Kennedy was explaining a solution to a possible issue with the original question?
a
Alexander Saydakov
10/02/2023, 6:33 PMthe topic starter might have a completely different issue
w
William Kennedy
10/02/2023, 6:36 PMyes I’m not too sure on the APPROX_COUNT_DISTINCT_DS_THETA behavior with input theta sketches in SQL, but definitely DS_THETA with input theta sketches in SQL just sketches the sketches and doesn’t work at all unless you include the undocumented COMPLEX_DECODE_BASE64 function information. As they mention getting the sketch into druid with DS_THETA is a probable workaround that they could test except it doesn’t work by default.
l
Lee Rhodes
10/02/2023, 6:44 PMWhat I am hearing is that there is no evidence, so far, that there is an accuracy issue with the DS_THETA sketch itself, but likely an issue with how it is being configured and used in the system. Is this a fair summary?
a
Alexander Saydakov
10/02/2023, 6:46 PMwe need more info from the topic starter
➕ 1