I am currently working on designing a pipeline for ML/Analytics in our infrastructure management software. We have historical data stored in Azure Data Lake, consisting of metric data presented as events and product logs from FFDC. In the initial phase, our objective is to analyze this metric data to identify anomalies and predictions such as memory prediction failure and usage patterns. For this task, we are considering using Apache Pinot, but I am curious if Apache Spark can also fulfill the same purpose. While I have watched a few YouTube videos by Chinmay on Pinot's real-time usage, I am interested in both , real-time analysis and ML for root cause analysis. Is there an ML library for Pinot similar to what Spark offers?

@Suvodeep Pyne ^^

Hey @Mahesh! ThirdEye might be able to help here. The usual workflow is to channel your metric data to pinot and set up thirdeye on top of it. You can setup alerts on thirdeye to monitor your data for anomalies and later do RCA on it. Regarding your question around ML, thirdeye ML is more geared towards anomaly detection. If your requirements are more around general ML then thirdeye may not be the best option. You still can host your own models and wire it up with thirdeye. However, I'd understand your use case first before recommending this option. Q: • You mentioned events/logs. What would be a good example of an anomaly here? Example: I'm looking for a drop in events, spike in logs, looking for a certain event type., etc • What's the data volume and at what granularity (daily, hourly, 15m, etc) do you want to monitor your data?

Hi @Suvodeep Pyne We cover all sorts of events, ranging from CPU, and memory failures to asset updates, additions/deletions, and issues like fan malfunctions, overheating, power drops, user addition/deletion, or RBAC activities, are captured by the infrastructure management software for BMC-based servers it oversees. There are 1000 to 5000 servers it manages at one data center. And we have a farm of such management software. Moreover, at intervals of every 20 seconds, it acquires metric data such as CPU, power, and memory statuses from each server it manages. For instance, rather than waiting for memory failure or the decline of server health, our aim is to proactively identify such anomalies well before they result in failure, reducing reliance solely on post-failure events.