Hi All, I am completely new to Apache Pinot and have some basic questions about whether this is the right fit for our use-case: 1. We have a network flow time-series data which we want to store 2. The data needs to be partitioned on a daily basis and if we see the same flow again within a day (imagine a flow is defined by a unique key probably), then we would just upsert the flow count for that in the same day. 3. There are applications which use this data to run adhoc queries for real-time analysis. Can someone with good background let me know if Pinot is a right fit for the above use-cases?

On the face of it, it is a good use cases. Would like to understand it in more details. For example, are you saying that rows should be upsertable only for 1 day?

if there’s no impact of performance when ingesting the data, then we are fine with upserting the data across all days too

Yeah, there’s should be no impact on performance. However, note that upsert only works with realtime tables. Is you data ingestion going to be via realtime?

yes, our data ingestion is going to be realtime

Upsert uses quite a lot of heap memory if you have lots of active keys e.g. daily date in the keys. You can achieve similar functionality using lastwithtime( col, timecol, 'double') and group by.