Hello all. I’m considering the use of pinot as a backend for a “business intelligence-like” tool for sensor data, that is: user customizable web facing dashboards + some reporting. I’m wondering if Pinot would be a good fit as my requirements are: • a derived metric definition mechanism that would allow developers to define metrics as a function of other metrics either timestamped or static • a fair amount of updates due to errors that might occur in the data transmission chain and due to data being ingested in batches • an “alternative data” mechanism that would set the value of a given metric A based on some criteria (like the existence of B for the same timestamp) • i need to handle approximately 1 million readings a day per production site. These metrics are transmitted each 5 minutes. Useful information can lie within this level of detail but the majority of the time, dashboards will present daily or hourly aggregates • each ingestion must trigger a set of analysis on the data that will perform actions based on business rules (like alerting, creating an issue ticket, etc...) I’m new to big data technologies and despite reading the documentation, I feel I’m missing a building block somewhere between postgresql and pinot. I’m thus seeking for friendly advice that could point me to the right direction. Thanks all.

thanks for providing detailed requirement. I can see Pinot has the primitives needed to achieve most of your requirements but for some of them you will to build something on top of Pinot

Thank you for your feedback. The web framework relies on a relational database for user management, and portfolio specifications (among others). With the concept of derived metrics, will I be able to define metrics based on data in the relational database?

We should probably take a concrete example. We can also move this conversation to troubleshooting channel