Hello, We're evaluating Apache Pinot for a fraud d...
# general
a
Hello, We're evaluating Apache Pinot for a fraud detection use case and wanted to get the community’s input on whether Pinot is a good fit, or if we’d need to pair it with something like Flink for stateful or more complex aggregations. Here’s what we’re trying to support: Aggregates • COUNT(), DISTINCT COUNT(), AVG(), MAX(), MIN(), SUM() • Statistical (nice to have not mandatory):
STDDEV()
,
Z-SCORE()
(for anomaly detection) • Time-based:
TIME_DIFF()
,
FIRST_SEEN()
,
LAST_SEEN()
• Geo:
DISTANCE()
between two coordinates,
TIMEZONE_DIFF()
Requirements • Support windowing for up to 12 months • Ingesting 10K events/sec • Query latency target: <200ms We know Pinot handles real-time ingestion well with fast OLAP queries, but for some of the stateful or time-windowed logic (e.g.,
Z-SCORE()
,
TIME_DIFF()
,
FIRST_SEEN()
), would it require using Flink or another stream processor, or can this be handled natively in Pinot (via UDFs, rollups, pre-aggregated tables, etc.) while still meeting requirements? Would love to hear any thoughts, lessons learned, or best practices from folks who’ve done similar Thanks in advance
m
Thanks for sharing the details. Pinot does support window functions natively since 1.0. However, given your read requirements you may want to precompute them.
n
Curious if Flink was chosen for this use-case or another stream processor 🙇‍♀️