Abdulaziz Alqahtani
07/23/2025, 10:09 AMSTDDEV(), Z-SCORE() (for anomaly detection)
• Time-based: TIME_DIFF(), FIRST_SEEN(), LAST_SEEN()
• Geo: DISTANCE() between two coordinates, TIMEZONE_DIFF()
Requirements
• Support windowing for up to 12 months
• Ingesting 10K events/sec
• Query latency target: <200ms
We know Pinot handles real-time ingestion well with fast OLAP queries, but for some of the stateful or time-windowed logic (e.g., Z-SCORE(), TIME_DIFF(), FIRST_SEEN()), would it require using Flink or another stream processor, or can this be handled natively in Pinot (via UDFs, rollups, pre-aggregated tables, etc.) while still meeting requirements?
Would love to hear any thoughts, lessons learned, or best practices from folks who’ve done similar
Thanks in advanceMayank
Neeraja Sridharan
10/31/2025, 3:23 AM