Hello everyone. I'm currently evaluating Apache Dr...
# general
j
Hello everyone. I'm currently evaluating Apache Druid for my use case. My use case is implementing Velocity Checks for fraud prevention.
Velocity checks monitor the number of times that certain transaction data elements occur within certain intervals and look for anomalies or similarities to known fraud behavior.
Examples • Looking at the number of transactions (velocity) by a card within a specified period of time (e.g., five transactions in 15 minutes) • Looking at the total amount for multiple transactions with a card in a specified period of time. I read that Druid is fast and can respond in milliseconds. Will the following queries be efficient in Druid? • How many transactions has a customer completed in the last 24 hours? • How much has a customer spent in the last 24 hours? How about if the time window range is in months instead of hours?
h
all of these queries can be answered by Druid efficiently. When it comes to how fast, Druid has near linear horizontal scaling. Optimizing the data (compaction) helps a lot too.
j
how about if the window is a year or more than a year?
h
druid breaks you data into segments as long as you provide enough cpu, the query can be parallelized and thus not slow down. Optimizing segments is a bit of a black art in Druid, don't pay attention to any benchmarks, unless the tester understands this. But even if you don't, Druid will give quite impressive response times.
My data goes back to 2018, we use a funnel to strike a balance between cluster cost and performance. I.e. the longer back, the less granular the data will be and the fewer dimensions we provide. All of this is highly use case specific. But you have all the flexibility do this at scale.
And by at scale I mean a continuous realtime intake of 35-45 million events per seconds for my use case. Again, near linear horizontal scaling, so (assuming a kafka intake) as long as you have enough kafka partitions, there is hardly a limit in Druid you will run into.