https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
a
could you clarify what do you mean by 70% deviation? from what? and what do you mean by “started” - did it work fine before? what has changed?
s
70% deviation: Consecutive query hitting gives sometimes 10k and sometimes 17k as a result “started”: Earlier the discrepancy between consecutive query results was not this much. Nothing has changed which should ideally impact queries over sketch
a
could you explain the use case please? what is the input? what is your query?
also what version of Druid are you running?
s
Basically, we ingest metrics about latency of a particular flow on the app. For example, if I’m doing something on the app and following a flow from page-1 to page-2 to page-3, likewise, we have an sdk which capture how much time it take for user to go through this flow.
Copy code
{
  "eventName": "flowName",
  "eventValueInMs": 100
}
A sample event like above gets ingested in druid and eventValue is a quantilesDoublesSketch. Now, a common queries over such type of data are like finding out p90. Putting a sample query below
Copy code
SELECT TIME_FLOOR(CAST("__time" AS TIMESTAMP), 'P1D') AS "__timestamp",
       APPROX_QUANTILE_DS(sketch_eventvalue, 0.9) AS "APPROX_QUANTILE_DS(sketch_eventvalue,0.9)"
FROM "druid"."garfield_dash_events"
WHERE "__time" >= '2023-07-04 00:00:00.000000'
  AND "__time" < '2023-07-11 00:00:00.000000'
  AND "namespaceid" IN ('APP-Android')
  AND "releasephase" IN ('RELEASE')
  AND "firstlaunch" IN (0)
  AND "tag2" IN ('NONE',
                 'NO_INTERRUPTION',
                 'FG_EXTERNAL')
  AND "eventname" = 'LAUNCH'
  AND "eventtype" = 'FLOW'
GROUP BY TIME_FLOOR(CAST("__time" AS TIMESTAMP), 'P1D')
LIMIT 5000;
Our current druid version is 0.19.0
a
wow, 0.19.0 is 3 years old
of course, the first question that comes to mind in such cases: can we reproduce the problem in the latest version? I understand that it might be difficult for you to answer that, but just in case you can upgrade your test environment or something, that would be my first suggestion
besides that, it would help to understand your distribution. and when you said that nothing has changed, did you mean that the distribution has not changed either? perhaps you started getting measurements with much wider spread or some outliers
another consideration is that the accuracy of the sketch is defined in the rank space, not in the space of values. when you get your p90 values, can you query the rank of them? what is the difference in rank?
another thing to consider is simplifying the complexity for debugging purposes. for instance, you could try getting a smallish set of raw measurements (say, a few million values) and play with them directly without Druid involved. say, create a sketch out of them in Python or something (there is a DataSketches Python module)
Open in Slack
PreviousNext
Powered by